Data processing method, electronic device, storage medium and program product

Abstract

The invention relates to the technical field of communication, and discloses a data processing method, electronic equipment, a storage medium and a program product. The data processing method provided by the invention is applied to the electronic equipment, and comprises the following steps: under the condition that first business data is obtained, splitting the first business data to obtain a preset number of business sub-data; the first business data is data generated by the electronic equipment in a business processing process; performing serialization processing on the service sub-data acquired in the first period to obtain first serialized data; compressing the first serialized data to obtain compressed data, and storing the compressed data in the electronic equipment; and uploading the compressed data to the server according to a second period. Thus, the electronic device can store the service data in the electronic device with a small occupied memory, the influence on the system performance of the electronic device is small, and the electronic device can still upload the data to the server in a weak network or network-free scene.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a data processing method, electronic equipment, storage medium and program product. Background Art

[0002] With the development of communication technology, car companies have developed applications (applications, APPs) (i.e., Internet of Vehicles business APPs) that can easily obtain vehicle information and control vehicle functions. Users can perform Internet of Vehicles business operations through Internet of Vehicles business APPs. In order to better serve car owners, Internet of Vehicles business APPs can obtain business data (such as buried point data) generated by users' Internet of Vehicles business operations and upload them to the server to optimize APP functions and improve user experience.

[0003] In actual car use scenarios, users usually park their vehicles in weak network or no network environments such as underground garages. When users use the Internet of Vehicles business APP in weak network or no network environment, the connection stability of the communication network between electronic devices and servers is relatively low. Therefore, the Internet of Vehicles business APP may be unable to upload business data to the server, resulting in the loss of business data, which may in turn affect the server's analysis and processing of business data. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present application provide a data processing method, an electronic device, a storage medium and a program product. The present application is introduced from multiple aspects below, and the implementation methods and beneficial effects of the following multiple aspects can be referenced to each other.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to an electronic device, and the method includes: when first business data is obtained, splitting the first business data to obtain a preset number of business sub-data; the first business data is data generated by the electronic device during the business processing process; serializing the business sub-data obtained within a first cycle to obtain first serialized data; compressing the first serialized data to obtain compressed data, and storing the compressed data in the electronic device; uploading the compressed data to the server according to a second cycle.

[0006] In the embodiment of the present application, the electronic device can split, serialize and compress the business data when receiving the business data, so that the data can be stored in the electronic device with a smaller memory usage, thus having a relatively small impact on the system performance of the electronic device. In addition, the electronic device can also periodically upload the stored data to the server, so that in scenarios where data cannot be reported due to weak network or no network, the electronic device can still upload the data to the server, thus avoiding data loss.

[0007] In a possible implementation of the first aspect, when the first business data is obtained, the first business data is split and processed to obtain a preset number of business sub-data, including: the electronic device splits and processes the first business data according to a preset data dimension to obtain a preset number of business sub-data; wherein each business sub-data corresponds to a data type, and at least some of the business sub-data have different data types.

[0008] It can be understood that in the embodiment of the present application, the business data is split to facilitate the subsequent processing of the business data by the electronic device or server. Exemplarily, the preset data dimensions may include data dimensions such as device information, timestamp, business scenario, business operation type, business parameters, and business processing results. The data type may include a long integer data type, a business enumeration type, a type-length-value (TLV) type, and the like.

[0009] In a possible implementation of the first aspect, the compressed data is uploaded to the server according to the second cycle, including: encrypting the compressed data using the first key to obtain encrypted data; signing the encrypted data based on a preset message authentication rule to obtain target transmission data; uploading the target transmission data to the server according to the second cycle.

[0010] In an embodiment of the present application, when an electronic device uploads data to a server, the data can be encrypted and signed. This can prevent the data from being tampered with or obtained by others during data transmission, thereby improving data security.

[0011] In a possible implementation of the first aspect, before encrypting the compressed data with the first key to obtain the encrypted data, the method also includes: obtaining the first key; when the first key is obtained and the first key is invalid, sending an authentication request to the server, so that the server sends a temporary public key to the electronic device when the electronic device is authenticated; generating a negotiated key based on the temporary public key, and using the negotiated key as the first key.

[0012] In an embodiment of the present application, the key used by the electronic device to encrypt data is generated after authentication of the server. When the server receives data uploaded by the electronic device, it can authenticate the identity of the electronic device based on the data, thereby further improving the security of data transmission.

[0013] In a possible implementation of the first aspect, the second cycle includes the current cycle and the next cycle after the current cycle; uploading the compressed data to the server according to the second cycle also includes: detecting the connection stability of the communication network between the server and the server during the current cycle; if the connection stability does not meet the preset conditions, pausing uploading the compressed data to the server in the current cycle and waiting for the next cycle.

[0014] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a server, and the method includes: receiving compressed data uploaded by an electronic device according to a second cycle; wherein the compressed data is obtained by the electronic device compressing first serialized data, and the first serialized data is stored in the electronic device; the first serialized data is obtained by the electronic device serializing business sub-data acquired in the first cycle; the business sub-data is obtained by the electronic device splitting the first business data when the first business data is acquired; the first business data is data generated by the electronic device during the business processing process.

[0015] In a possible implementation of the second aspect, the server includes a first load balancing cluster, a distributed message cluster and a log processing service; the method includes: when the first load balancing cluster obtains compressed data uploaded by the electronic device, the compressed data is written into a first log file; the log processing service collects and parses the first log file to obtain a preset number of business sub-data, and sends the preset number of business sub-data to the distributed message cluster.

[0016] In an embodiment of the present application, after the server obtains the data uploaded by the electronic device through the first load balancing cluster, the data is first written into a log file, and then the content in the log file is collected and parsed. In this way, asynchronous process processing of the data is realized, the data processing performance of the server can be improved, and the server can realize high-throughput data collection.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more programs; and the processor is used to execute the one or more programs so that the electronic device implements the data processing method in the above-mentioned first aspect or second aspect and any possible implementation of the first aspect or the second aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the data processing method in the above-mentioned first aspect or second aspect and any possible implementation of the first aspect or second aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a program product, which, when executed on an electronic device, enables the electronic device to implement the data processing method in the above-mentioned first aspect or second aspect and any possible implementation of the first aspect or second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 According to some embodiments of the present application, a flow chart of a data processing method is shown;

[0021] Figure 2 According to some embodiments of the present application, a schematic diagram of a process in which a device side requests authentication from a server side is shown;

[0022] Figure 3 According to some embodiments of the present application, a schematic diagram of a process of uploading business data from a device to a server is shown;

[0023] Figure 4 According to some embodiments of the present application, a schematic structural diagram of an electronic device is shown. DETAILED DESCRIPTION

[0024] The illustrative embodiments of the present application include but are not limited to a data processing method, an electronic device, a storage medium and a program product. The method of the present application is introduced below in conjunction with specific embodiments.

[0025] In order to more clearly understand the solution of the present application, the relevant field terms involved in the present application are first explained.

[0026] Buried data: Buried data refers to various behavioral data collected during the user's use of an application or website after a mark is buried in the application or website. Behavior data may include, for example, clicks, slides, searches, etc. In the embodiment of the present application, the buried data is the business operation data generated by the user's business operations on the Internet of Vehicles business APP, where the business operations may include near-field vehicle control, remote vehicle control, vehicle status information query, and other business operations.

[0027] As mentioned above, when users use the Internet of Vehicles business APP in a weak network or no network environment such as an underground garage, the connection stability of the communication network between the electronic device and the server is relatively low. Therefore, the Internet of Vehicles business APP may not be able to securely report the business data generated by the user's Internet of Vehicles business operations to the server, resulting in the loss of business data, which may in turn affect the server's analysis and processing of business data.

[0028] In view of this, a data processing method is provided in an embodiment of the present application. In this method, after obtaining the business data, the electronic device can first split the business data into fields to obtain business sub-data, then serialize the business sub-data obtained in the first cycle, compress and store them, and then upload the business data to the server according to the second cycle.

[0029] In this way, the data processing method provided in the embodiment of the present application can first split, serialize and compress the business data when receiving the business data, and store it in the electronic device with a smaller memory footprint, so that the impact on the system performance of the electronic device is relatively small. Since the data has been stored in the electronic device, in the scenario where there is no network or a weak network and data cannot be reported, the electronic device can still upload the data to the server, and the problem of data loss can also be avoided.

[0030] In some embodiments, when a user's business operation on an application in an electronic device (such as an Internet of Vehicles business APP) is detected, the electronic device can generate business data (such as buried point data) corresponding to the business operation, and can obtain and process the business data.

[0031] Combine the following Figure 1 The flowchart shown in FIG. 1 introduces a data processing method provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0032] S101: When the electronic device obtains first business data, the first business data is split and processed to obtain a preset number of business sub-data.

[0033] In some embodiments, when the electronic device obtains the first business data, it splits the first business data to obtain a preset number of business sub-data, wherein the first business data is data generated by the electronic device during the business processing.

[0034] Exemplarily, in a vehicle networking scenario, the first business data may be business data generated by an electronic device in response to a user's business operation on a vehicle networking business APP, for example, buried point data.

[0035] In some embodiments, when the electronic device obtains the first business data, it splits the first business data according to a preset data dimension to obtain a preset number of business sub-data, where each business sub-data corresponds to a data type, and at least some of the business sub-data have different data types.

[0036] For example, for the Internet of Vehicles scenario, the preset data dimensions may include vehicle identification number (VIN), device identifier (i.e., device ID), device information, timestamp, business scenario, business operation type, business parameters, business processing results, additional data, etc. In addition, the data type of the VIN code is character type, the timestamp is long integer data type, the business scenario and business operation are both business enumeration types, and the business parameters, business processing results, and additional data are TLV types.

[0037] It can be understood that the preset data dimensions may include more or fewer dimensions than those in the above examples, and may also include other data dimensions. The specific settings can be made according to actual needs, and this application does not limit this.

[0038] S102: The electronic device serializes the service sub-data acquired in the first cycle to obtain first serialized data.

[0039] In some embodiments, the electronic device serializes the business sub-data acquired in the first cycle to obtain first serialized data.

[0040] It can be understood that the service sub-data acquired in the first cycle is obtained by the electronic device after splitting and processing multiple first service data acquired in the first cycle. The first cycle can be set according to actual application requirements, and this application does not limit the length of the first cycle.

[0041] Exemplarily, taking the first cycle as 5 seconds, and the electronic device obtaining 50 first business data within the first cycle as an example, the electronic device can split the 50 first business data obtained within 5 seconds, serialize the split business sub-data, and obtain first serialized data.

[0042] In some embodiments, the electronic device may use a structured data serialization method, such as a protobuf method, to serialize the business sub-data. The protobuf method is a language-independent, platform-independent, and extensible method for serializing structured data, which can convert a data structure or object into a format that can be stored or transmitted.

[0043] S103: The electronic device compresses the first serialized data to obtain compressed data, and stores the compressed data in the electronic device.

[0044] In some embodiments, after obtaining the first serialized data, the electronic device compresses the first serialized data to obtain compressed data, and stores the compressed data in the electronic device.

[0045] Specifically, the electronic device may use a compression algorithm to compress the first serialized data to obtain compressed data. Exemplarily, the compression algorithm may be a gzip compression algorithm. After the electronic device obtains the compressed data, it may store the compressed data in a local cache of the electronic device. Exemplarily, if the compressed data corresponds to APP1, the electronic device may store the compressed data in a local cache of APP1.

[0046] S104, the electronic device uploads the compressed data to the server according to the second cycle.

[0047] In some embodiments, the electronic device can upload the stored compressed data to the server according to the second period. It can be understood that the second period can be set according to actual application requirements, and this application does not limit this.

[0048] In some embodiments, the electronic device may set a timed task with a second period (for example, the second period may be 1 second), obtain compressed data from a local cache according to a first-in-first-out principle, and upload the compressed data to a server using a weak network transmission protocol. Exemplarily, the weak network transmission protocol may be a quick UDP internet connections (QUIC) protocol.

[0049] It is understandable that when electronic devices upload business data, they follow the first-in-first-out principle, that is, the first generated business data will be uploaded to the server first, which can avoid data loss, simplify data processing logic, and reduce implementation complexity. In addition, using a weak network transmission protocol to upload data can ensure that the data can be successfully uploaded to the server in a weak network scenario.

[0050] In some embodiments, the electronic device uses the first key to encrypt the compressed data to obtain encrypted data. Then, the encrypted data is signed based on the preset message authentication rule to obtain the target transmission data, and the target transmission data is uploaded to the server according to the second cycle, wherein the target transmission data may include the encrypted data and the signature. The preset message authentication rule may be a hash-based message authentication code (Hmac) algorithm, such as the HmacSHA256 algorithm.

[0051] Exemplarily, the electronic device may utilize the first key, and use the advanced encryption standard (AES) algorithm, the cipher block chaining (CBC) mode, and the PKCS#5 padding (PKCS5Padding) to encrypt the compressed data.

[0052] It can be understood that encrypting the data when the electronic device uploads it to the server can prevent the data from being tampered with and prevent other users from obtaining the data, thereby ensuring the security of the data during transmission. In addition, after the electronic device signs the data, the server can verify the identity of the electronic device based on the signature in the target transmission data, thereby further ensuring the security of data transmission.

[0053] In some embodiments, after the electronic device obtains the encrypted data, it can perform a signature process on the original data based on a preset message authentication rule, and the obtained target transmission data includes the encrypted data and the signature.

[0054] In some embodiments, before the compressed data is encrypted using the first key to obtain the encrypted data, the electronic device obtains the first key. When the first key is obtained and the first key is invalid, or when the first key is not obtained (that is, the first key does not exist), the electronic device sends an authentication request to the server, so that the server sends a temporary public key to the electronic device when the electronic device is authenticated. When the electronic device receives the temporary public key, it generates a negotiated key based on the temporary public key and uses the negotiated key as the first key. The process of the electronic device requesting authentication from the server will be described in detail below and will not be repeated here.

[0055] In some embodiments, the second cycle includes the current cycle and the next cycle after the current cycle, and the electronic device uploads the compressed data to the server according to the second cycle (for example, 1 second). Before uploading the compressed data in the current cycle, the electronic device first detects the connection stability of the communication network between the electronic device and the server in the current cycle. If the connection stability does not meet the preset conditions, the uploading of compressed data to the server is suspended in the current cycle and waits for the next cycle. Among them, the preset conditions may include the network bandwidth being higher than the first threshold (for example, 10KB / s), or the network delay being lower than the second threshold (for example, 200 milliseconds), or the packet loss rate being lower than the third threshold (for example, 0.5%), etc.

[0056] It can be understood that when the connection stability of the communication network between the electronic device and the server does not meet the preset conditions in the current cycle, that is, the network connection stability is relatively low, it can correspond to a no-network scenario. Similarly, when the network connection stability is relatively high, it can correspond to a weak network or a network scenario. In a no-network scenario, the connection stability of the communication network between the electronic device and the server is relatively low, and the electronic device usually cannot successfully communicate with the server, and thus cannot successfully upload data to the server.

[0057] In some embodiments, a server receives compressed data uploaded by an electronic device according to a second cycle; wherein the compressed data is obtained by the electronic device compressing first serialized data, and the first serialized data is stored in the electronic device, the first serialized data is obtained by the electronic device serializing business sub-data acquired in the first cycle, the business sub-data is obtained by the electronic device splitting the first business data when the first business data is acquired, and the first business data is data generated by the electronic device during business processing.

[0058] It can be understood that with the continuous development of the number of Internet of Vehicles users and Internet of Vehicles services, the amount of data of electronic devices (such as the amount of data of buried point data) and the reported access volume of servers have increased dramatically. Correspondingly, the server faces the challenge of high throughput, low latency and high stability data collection. In order to cope with this scenario, in some embodiments, the server includes a first load balancing cluster, a distributed message cluster and a log processing service, wherein the first load balancing cluster is used to reduce the pressure of the server processing requests and improve the processing power and reliability of the server. Exemplarily, the first load balancing cluster can be an Nginx cluster. The distributed message cluster is used to build a high-throughput, scalable message system, which can improve the reliability of data. Exemplarily, the distributed message cluster can be a Kafka cluster. The log processing service is used to collect log data and analyze it in real time. Exemplarily, the log processing service can be a flume component.

[0059] It can be understood that, taking the first load balancing cluster as the Nginx cluster as an example, by increasing the nodes of the Nginx cluster, the query rate per second (QPS) of the server can be increased, that is, the amount of request data processed per unit computing resources per unit time, so that the server can meet the data collection requirements of high throughput, low latency and high stability.

[0060] In some embodiments, when the first load balancing cluster in the server obtains the compressed data uploaded by the electronic device, the compressed data is written into the first log file. The log processing service collects and parses the first log file to obtain a preset number of business sub-data, and sends the preset number of business sub-data to the distributed message cluster.

[0061] Specifically, after the log processing service collects the first log file, it analyzes the content in the first log file and sends the obtained data to the distributed message cluster, wherein the compressed data may be compressed data after encryption and signature processing, and the data obtained after the log processing service analyzes the first log file may include a signature.

[0062] It can be understood that the server first writes the data uploaded by the electronic device into a log file, and then obtains the data uploaded by the electronic device by collecting and parsing the log file and sends the data to the distributed message cluster, thereby realizing the decoupling of data collection and parsing. In this way, even if all server components except the first load balancing cluster are down, the server can continue to collect data uploaded by the electronic device, thus avoiding data loss.

[0063] In some embodiments, the server may further include a data analysis service. When the log processing service sends data to the distributed message cluster, the data analysis service may obtain the encrypted data and signature from the distributed message cluster, and perform decryption processing, signature verification processing, decompression processing, deserialization processing, and field analysis processing to obtain a preset number of business sub-data. Then, the preset number of business sub-data is sent to the distributed message cluster.

[0064] In some embodiments, the server may also include a second load balancing cluster, which is used to receive data uploaded by the electronic device and send the received data to the first load balancing cluster, wherein the second load balancing cluster may be, for example, an LVS (linux virtual server) load balancing cluster.

[0065] The server in the embodiment of the present application can be a cloud server, a physical server, etc., such as an X86 server, an ARM server, etc. It can also be a virtual machine (VM) based on a general physical server combined with network function virtualization (NFV) technology. A virtual machine refers to a complete computer system with complete hardware system functions and running in a completely isolated environment simulated by software. It can also be application software, which can run on a cloud server or other computing devices. This application does not impose any restrictions on the specific type of server.

[0066] Combine the following Figure 2 , introduces the process of the electronic device requesting authentication from the server mentioned in S104. For the sake of understanding, the following description is made by taking the cloud server as an example.

[0067] Reference Figure 2 In some embodiments, as Figure 2As shown, when the electronic device obtains the first key and the first key is invalid, or when the first key is not obtained (that is, the first key does not exist), the electronic device sends an authentication request to the cloud server. Before sending the authentication request, the electronic device pre-prepare the device private key of the device in advance, and encrypts and stores the private key, for example, using a local obfuscation strategy or a third-party secure white box storage strategy. In addition, the cloud server pre-prepare the device public key corresponding to the private key in advance and securely stores it on the server side.

[0068] It can be understood that encrypting and storing the private key can ensure the security of the private key.

[0069] In some embodiments, the local storage of the electronic device stores the device ID and authentication timestamp of the electronic device. Before the electronic device sends an authentication request to the server, a temporary public-private key pair is first generated locally, and the temporary public-private key pair includes a temporary private key 1 and a temporary public key 2. When the electronic device sends an authentication request, the authentication timestamp is updated, and the device ID, authentication timestamp and temporary public key 2 are used as parameters. The parameters are then signed using the device private key and a digital signature algorithm to generate a parameter signature S1. Among them, the authentication timestamp can be the moment when the electronic device sends an authentication request to the cloud server, and the digital signature algorithm can be, for example, the SHA256withECDSA algorithm. The SHA256withECDSA algorithm is a digital signature scheme that combines the elliptic curve digital signature algorithm (ECDSA) and the secure hash algorithm (SHA-256), and has relatively high security.

[0070] In some embodiments, after the electronic device signs the parameters, it generates an authentication request based on the parameters and the parameter signature S1, and sends the authentication request to the cloud server.

[0071] Continue to refer to Figure 2In some embodiments, the cloud server includes an authentication service, and the authentication service includes a remote dictionary service (Redis). When the authentication service in the cloud server receives an authentication request, it first uses the pre-prepared device public key to verify the parameter signature S1 in the authentication request, and verifies whether the time difference between the authentication timestamp in the authentication request and the cloud timestamp (i.e., the time when the cloud server receives the authentication request) is greater than a fourth threshold value (e.g., 5 minutes). If the time difference is greater than the fourth threshold value, the authentication process is interrupted and an error reminder is returned to the electronic device to remind the electronic device to calibrate the device time. If the time difference is less than the fourth threshold value and the device public key passes the verification of the parameter signature S1, the cloud server determines that the authentication of the electronic device passes and generates a temporary public-private key pair, which includes a temporary private key 3 and a temporary public key 4. Then, the cloud server uses the temporary private key 3 and the temporary public key 2 in the authentication request for key negotiation to generate a second key (i.e., a temporary symmetric key A, referred to as key A) and the validity period of key A. Afterwards, the cloud server stores key A, the validity period of key A, the device ID and the authentication timestamp in the authentication request in the form of a key-value pair (Key: Value) in Redis, where Key = device ID + authentication timestamp, Value = key A + validity period. In addition, the cloud server returns the authentication result to the electronic device, which may include the temporary public key 4 and the validity period of key A.

[0072] Exemplarily, when the cloud server uses the temporary private key 3 and the temporary public key 2 in the authentication request to perform key negotiation to generate the key A, the ECDH (elliptic curve diffie-hellman) key negotiation algorithm can be used.

[0073] In some embodiments, when the electronic device receives the authentication result from the cloud server, it uses the temporary public key 4 in the authentication result to negotiate with the temporary private key 1 stored in the third-party security white box to generate a first key (i.e., temporary symmetric key B, referred to as key B). The electronic device uses the validity period of key A as the validity period of key B, and encrypts and stores key B, the authentication timestamp, and the validity period of key B, for example, using a local obfuscation strategy or a third-party security white box storage strategy for storage.

[0074] Exemplarily, when the electronic device uses the temporary public key 4 and the temporary private key 1 stored in the third-party security white box to perform key negotiation to generate the key B, the ECDH key negotiation algorithm may also be used.

[0075] It can be understood that key B stored in the electronic device is used to encrypt business data, and key A stored in the cloud server is used to decrypt data uploaded by the electronic device and encrypted by key B. Therefore, key A corresponds to key B, and key A and key B have the same validity period.

[0076] It can be understood that in the scenario of uploading buried data, users are usually not required to log in to the Internet of Vehicles APP. However, when the user is not logged in, the Internet of Vehicles APP lacks a secure reporting mechanism when uploading business data, and the cloud server may not be able to confirm the authenticity and integrity of the business data. In an embodiment of the present application, when the electronic device uploads business data, it uses the first key generated after authentication by the cloud server to encrypt the business data. After the cloud server receives the business data uploaded by the electronic device, it can use the second key corresponding to the first key to decrypt the business data. In this way, the security of data transmission is improved by increasing the verification mechanism of data credibility.

[0077] For the convenience of description, the following takes the Internet of Vehicles scenario as an example. Figure 3 The schematic diagram of the process of uploading business data from the device to the server shows a data processing method provided in an embodiment of the present application.

[0078] Reference Figure 3 In some embodiments, when the electronic device detects a vehicle networking service operation, such as near-field vehicle control, remote vehicle control, vehicle status information query, etc., the service data generated by the service operation is split into fields. For example, the fields are split according to nine data dimensions: VIN code, device ID, mobile device information, timestamp, service scenario, service operation type, service parameter, service processing result, and additional data, where the data type of each data dimension is consistent.

[0079] In some embodiments, after the electronic device splits the business data, the business sub-data obtained after the periodic accumulation and splitting are serialized and compressed. Specifically, the electronic device takes the business sub-data obtained in the first cycle as a batch, performs serialization processing, and obtains serialized data. Then, after compressing the serialized data, compressed data is obtained, and the compressed data is placed in the cache pool.

[0080] For example, the electronic device can serialize the business sub-data using the protobuf method to obtain serialized data, and then compress the serialized data using the gzip compression algorithm to obtain compressed data and put it into the cache pool. Figure 3 As shown, the data in the cache pool includes data1, data2, data3, data4, etc. in the order in which the compressed data is generated.

[0081] In some embodiments, when the electronic device uploads data to the cloud server, it periodically extracts the compressed data in the cache pool according to the first-in-first-out principle. Specifically, the electronic device extracts the compressed data in the cache pool according to the second period for uploading. Exemplarily, the second period can be 1 second.

[0082] It is understandable that the second period can be set according to actual application requirements, and this application does not impose any restrictions on this.

[0083] In some embodiments, after the electronic device extracts the compressed data from the cache pool, it encrypts the compressed data. Specifically, the electronic device first checks the validity of the key B stored locally in the security white box. When it is determined that the key B is valid, the electronic device uses the key B to encrypt the compressed data (for example, data4), and after obtaining the encrypted data, it uses the preset message authentication rules to sign the encrypted data to obtain the target transmission data. After that, the electronic device performs a network status check, that is, detects the connection stability of the communication network between the electronic device and the cloud server in the current cycle. When the electronic device determines that the connection stability meets the preset conditions, the electronic device uses the QUIC protocol to call the cloud upload interface to upload the target transmission data to the cloud server. Among them, the target transmission data may include encrypted data, signature, authentication timestamp and device ID.

[0084] Exemplarily, the electronic device may encrypt the compressed data using the Advanced Encryption Standard (AES) algorithm, the Cipher Block Chaining (CBC) mode, and the PKCS#5 padding (PKCS5Padding) using the key B. The preset message authentication rule may be the HmacSHA256 algorithm.

[0085] It can be understood that encrypting the data when the electronic device uploads the data to the server can prevent the data from being tampered with and prevent other users from obtaining the data, thereby ensuring the security of the data during transmission.

[0086] In some embodiments, when the electronic device determines that the connection stability does not meet a preset condition, the electronic device suspends uploading data to the cloud server in the current cycle and waits for the next cycle.

[0087] It can be understood that in the next cycle, the electronic device can continue to upload the data suspended in the current cycle, such as the target transmission data obtained after encrypting and signing data4, without extracting the compressed data from the cache pool again.

[0088] In an embodiment of the present application, when the electronic device uploads data to a cloud server, it may first encrypt and sign the compressed data and then perform a network status check, or it may first perform a network status check and then encrypt and sign the compressed data. The present application does not impose any restrictions on this.

[0089] In some embodiments, when the electronic device determines that key B is invalid or does not exist, the electronic device sends an authentication request to the cloud server. When the cloud server authenticates the electronic device, key A is generated and stored in the Redis of the cloud server, and the cloud server returns a temporary public key 4 to the electronic device. After the electronic device receives the temporary public key 4, it generates key B based on the temporary public key 4. The specific authentication process can be referred to the above Figure 2 The relevant description will not be repeated here.

[0090] In some embodiments, when the cloud server receives the target transmission data, it can return a response message indicating successful data reception to the electronic device. When the electronic device receives the response message, it deletes the uploaded data.

[0091] Continue to refer to Figure 3 In some embodiments, the cloud server includes an LVS load balancing cluster, an Nginx cluster, an authentication service, a Kafka cluster, a log processing service, a data storage service, a data analysis service, and a storage middleware.

[0092] The LVS load balancing cluster is used to receive data sent by electronic devices and forward the data to the Nginx cluster.

[0093] The Nginx cluster is used to receive data sent by LVS and write the data to log files, namely, Nginx log files.

[0094] The authentication service is used to authenticate the electronic device based on the authentication request sent by the electronic device, and return the authentication result to the electronic device.

[0095] The Kafka cluster can be used as a data pipeline to achieve asynchronous data transmission and real-time data processing.

[0096] The log processing service includes an output component (Sink), a storage component (Channel) and an input component (Source), which are used to collect and parse the content in the log file and send the parsed content to the Kafka cluster. Exemplarily, the log processing service can be a flume component.

[0097] The data parsing service is used to obtain data from the Kafka cluster and parse it, including decryption, signature verification, decompression, deserialization, field parsing, etc.

[0098] The data storage service is used to obtain data from the Kafka cluster, convert the data format, and store the data in the storage middleware.

[0099] The storage middleware includes a data warehouse tool (Hive), which is used to store data and provide a data source for IoV data analysis.

[0100] In some embodiments, when the cloud server receives the target transmission data, the target transmission data is routed to the Nginx cluster through the LVS load balancing cluster, and the Nginx cluster writes the target transmission data into the Nginx log file. In addition, the log processing service collects and parses the Nginx log file to obtain the encrypted data, signature, device ID and authentication timestamp, and sends the encrypted data, signature, device ID and authentication timestamp to the Kafka cluster.

[0101] In some embodiments, after the log processing service collects the Nginx log file and parses the file content, the obtained data may also include an upload timestamp, that is, the time when the electronic device records the data upload.

[0102] It can be understood that the Nginx cluster receives the target transmission data and writes it to the log file. The log processing service collects and parses the log file, obtains the data uploaded by the electronic device and sends the data to the distributed message cluster, thereby decoupling data collection and parsing. In this way, when all server components except the LVS load balancing cluster and the Nginx cluster are down, the cloud server can continue to collect data uploaded by the electronic device, thus avoiding data loss.

[0103] Continue to refer to Figure 2 and Figure 3 In some embodiments, the data parsing service can obtain the encrypted data, signature, device ID and authentication timestamp from the Kafka cluster, and use the device ID and authentication timestamp as keys to search for the key A and validity period corresponding to the encrypted data in the Redis of the authentication service. After determining the corresponding key A and validity period, determine whether key A has expired. If key A has expired, discard key A. If key A has not expired, use key A to decrypt the encrypted data, and then verify the signature of the decrypted data.

[0104] Exemplarily, the data analysis service can use key A to decrypt the compressed data using the Advanced Encryption Standard (AES) algorithm, Cipher Block Chaining (CBC) mode, and PKCS#5 padding (PKCS5Padding). In addition, the data analysis service can verify the signature of the decrypted data using the HmacSHA256 algorithm.

[0105] It can be understood that in order to ensure that the cloud server can restore the business data uploaded by the electronic device, the algorithm used by the cloud server to decrypt the data should correspond to the algorithm used by the electronic device to encrypt the business data, and the algorithm used by the cloud server to verify the data should correspond to the algorithm used by the electronic device to sign the business data.

[0106] In some embodiments, when the signature verification passes, the data parsing service continues to decompress the decrypted data, deserialize the decompressed data, and then performs field parsing on the deserialized data, and finally sends the data obtained after the field parsing to the Kafka cluster.

[0107] Exemplarily, the data parsing service may decompress the decrypted data using a gzip algorithm, and deserialize the decompressed data using a protobuf method.

[0108] It can be understood that in order to ensure that the cloud server can restore the business data uploaded by the electronic device, the cloud server's data decompression algorithm should correspond to the algorithm used by the electronic device to compress the business data, and the cloud server's data deserialization algorithm should correspond to the algorithm used by the electronic device to serialize the business data.

[0109] In some embodiments, the data storage service can obtain data from the Kafka cluster (i.e., data obtained through field parsing), convert the data into Hive format, and store the converted data in batches in Hive to provide a data source for subsequent Internet of Vehicles data analysis.

[0110] In this way, the data processing method provided by the embodiment of the present application can enable the electronic device to store the business data in the electronic device with a smaller memory footprint when receiving the business data, and can periodically upload the stored business data to the server, so that for weak network or no network scenarios, since the data has been stored, the problem of data loss can be avoided. In addition, before the electronic device uploads the business data, it has obtained the server's authentication and encrypted the business data, thereby improving the security and reliability of the data. On the server side, through the strategy of asynchronous collection and processing decoupling, a series of operations such as data collection, parsing, verification, and storage are streamlined, and data can be collected to the server safely, efficiently, and with high throughput.

[0111] An embodiment of the present application also provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the data processing method mentioned in the present application.

[0112] The present application also provides a program product. When the program product is executed on an electronic device, the electronic device implements the data processing method mentioned in the present application.

[0113] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more programs; and the processor is used to execute the one or more programs so that the electronic device implements the data processing method mentioned in the present application.

[0114] Combine the following Figure 4 , introduces an electronic device provided in an embodiment of the present application.

[0115] like Figure 4 As shown, the electronic device may include a transceiver 411 , a processor 412 , and a memory 413 .

[0116] The transceiver 411 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0117] The processor 412 executes the computer-executable instructions stored in the memory, so that the processor 412 executes the scheme in the above embodiment. The processor 412 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital data processor DSP, an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0118] The memory 413 is connected to the processor 412 via a system bus to complete the communication between them. The memory 413 is used to store computer program instructions.

[0119] By way of example and not limitation, the memory 413 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 413 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 413 may be inside or outside the integrated gateway device. In some embodiments, the memory 413 is a non-volatile solid-state memory. In some embodiments, the memory 413 includes a read-only memory (ROM). Where appropriate, the read-only memory may be a mask-programmed read-only memory, a programmable read-only memory (PROM), an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory, or a combination of two or more of these.

[0120] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0121] The method provided in the embodiment of the present application can be applied to any electronic device, including but not limited to a mobile station (MS), a mobile terminal (MT), etc. For example, the electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a desktop computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, etc. The embodiment of the present application does not limit the specific form of the electronic device.

[0122] The various embodiments disclosed in the present application may be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0123] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit, or a microprocessor.

[0124] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0125] In some cases, the disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory, a random access memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic or other form of propagation signal. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0126] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0127] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0128] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprise one" do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0129] Although the present application has been illustrated and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to electronic equipment, the method comprises: When the first service data is acquired, the first service data is split and processed to obtain a preset number of service sub-data; the first service data is data generated by the electronic device during service processing; Serializing the business sub-data acquired in the first cycle to obtain first serialized data; Compressing the first serialized data to obtain compressed data, and storing the compressed data in the electronic device; The compressed data is uploaded to the server according to the second cycle.

2. The method according to claim 1, characterized in that When the first business data is obtained, the first business data is split to obtain a preset number of business sub-data, including: The electronic device splits the first service data according to a preset data dimension to obtain a preset number of service sub-data; Each of the service sub-data corresponds to a data type, and at least some of the service sub-data have different data types.

3. The method according to claim 1, characterized in that The step of uploading the compressed data to the server according to the second cycle includes: Encrypting the compressed data using the first key to obtain encrypted data; Performing a signature process on the encrypted data based on a preset message authentication rule to obtain target transmission data; The target transmission data is uploaded to the server according to the second cycle.

4. The method according to claim 3, characterized in that Before encrypting the compressed data with the first key to obtain the encrypted data, the method further includes: Obtaining the first key; When the first key is acquired and is invalid, sending an authentication request to the server, so that the server sends a temporary public key to the electronic device when the electronic device is authenticated; A negotiated key is generated according to the temporary public key, and the negotiated key is used as the first key.

5. The method according to claim 3, characterized in that: The second cycle includes a current cycle and a next cycle after the current cycle; The step of uploading the compressed data to the server according to the second cycle further includes: Detecting the connection stability of the communication network with the server during the current period; When the connection stability does not meet the preset condition, uploading the compressed data to the server is suspended in the current cycle and waiting for the next cycle.

6. A data processing method, characterized in that: Applied to a server, the method comprises: Receiving compressed data uploaded by the electronic device according to the second cycle; wherein the compressed data is obtained by the electronic device compressing the first serialized data, and the first serialized data is stored in the electronic device; The first serialized data is obtained by the electronic device performing serialization processing on the service sub-data acquired in the first cycle; The business sub-data is obtained by splitting the first business data when the electronic device obtains the first business data; the first business data is the data generated by the electronic device during the business processing.

7. The method according to claim 6, characterized in that The server includes a first load balancing cluster, a distributed message cluster and a log processing service; The method comprises: When the first load balancing cluster obtains the compressed data uploaded by the electronic device, writing the compressed data into a first log file; The log processing service collects and parses the first log file to obtain a preset number of business sub-data, and sends the preset number of the business sub-data to the distributed message cluster.

8. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store one or more programs; The processor is used to execute the one or more programs so that the electronic device implements the data processing method according to any one of claims 1 to 5 or claims 6 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the data processing method according to any one of claims 1 to 5 or claims 6 to 7.

10. A program product, characterized in that When the program product is executed on an electronic device, the electronic device implements the data processing method described in any one of claims 1 to 5 or claims 6 to 7.

Images