Cloud data processing method and device, terminal equipment and storage medium

Abstract

The invention provides a cloud data processing method and device, terminal equipment and a storage medium, and the method comprises the steps: constructing a client connection pool at a cloud, connecting a plurality of clients based on the client connection pool, and dividing the clients into a plurality of shared subscription groups; wherein each shared subscription group correspondingly receives vehicle data of one theme; the control server calls a corresponding shared subscription group according to the theme type of the received vehicle data, and randomly sends the vehicle data to any client in the shared subscription group; wherein the server is used for receiving vehicle data uploaded by a vehicle end; and sending the vehicle data received by the client to a pre-constructed thread pool for asynchronous processing. According to the invention, the processing capability of high-concurrency data can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing and distributed computing, and particularly to a method, device, terminal device and storage medium for processing cloud data. Background Art

[0002] The cloud, abbreviated as "cloud", refers to the mode of addition, use and interaction of Internet-based related services, usually involving providing dynamically scalable and often virtualized resources through the Internet. Cloud computing is to transfer data, software and processing capabilities to a platform located on a remote server, enabling users to access these services through any Internet-connected device. Main features such as cloud storage, elastic computing capabilities, on-demand services, multi-tenant architecture, and high-speed data processing capabilities have made cloud computing widely used in data-intensive scenarios, such as big data analysis, machine learning, and artificial intelligence. In the field of vehicle data processing, the combination of the cloud and the client has become a mainstream method. Specifically, a vehicle (client) can collect various running data, such as location information, speed, fuel consumption, driving habits, etc., and upload this data to the cloud. The cloud can summarize and analyze the data to mine valuable information, such as generating a driving behavior analysis report, predicting vehicle maintenance time, and optimizing the driving route.

[0003] In the implementation of processing vehicle data by combining the cloud and the client, in the single-client design mode, after the system runs for a period of time, a single client needs to process a large amount of real-time data (such as location, status, instructions, etc.) sent by the vehicle. As the amount of data increases, the computing resources and processing capabilities of the client may gradually be exhausted, resulting in untimely message processing and even system crashes. The single-client design lacks redundancy and load balancing mechanisms and cannot effectively handle high-concurrency data processing requirements. Moreover, vehicle data processing involves the transmission and calculation of a large amount of real-time data. If the processing capabilities of the client are insufficient or the data processing logic is not designed reasonably, it may lead to data backlog, further exacerbating the problem of untimely message processing. For example, the message queue may not be effectively managed, resulting in backlogged messages that cannot be processed in time, ultimately causing the vehicle scheduling system to crash. Summary of the Invention

[0004] The present invention aims to provide a method, device, terminal device and storage medium for processing cloud data to solve the above technical problems and improve the processing ability for high-concurrency data.

[0005] To solve the above technical problems, the present invention provides a method for processing cloud data, including:

[0006] Build a client connection pool in the cloud, connect several clients based on the client connection pool, and divide the clients into several shared subscription groups; where each of the shared subscription groups corresponds to receiving vehicle data of one topic.

[0007] The control server calls the corresponding shared subscription group according to the topic type of the received vehicle data, and randomly sends the vehicle data to any one of the clients within the shared subscription group; where the server is used to receive the vehicle data uploaded by the vehicle terminal.

[0008] Send the vehicle data received by the client to a pre-built thread pool for asynchronous processing.

[0009] In the above solution, by building a client connection pool, the client connections can be effectively managed and allocated, avoiding overloading of a single client. Dividing the clients into multiple shared subscription groups, with each group corresponding to vehicle data of one topic, can achieve load balancing and ensure that the clients within each shared subscription group can share the processing tasks. The clients within each shared subscription group can back up each other. Even if a certain client fails, other clients can still continue to process the data, improving the high availability of the system. Sending the vehicle data received by the client to a pre-built thread pool for asynchronous processing improves the efficiency and response speed of data processing, and avoids the blocking and delay that may be caused by synchronous processing. The thread pool can effectively manage multi-threaded tasks, optimize resource utilization, and improve the processing capacity of the system. Through the combination of shared subscription groups and the client connection pool, even if a certain client fails, the system can still continue to run, ensuring the continuity of data processing.

[0010] In one implementation, the building of the client connection pool in the cloud specifically includes:

[0011] Create a cloud configuration file, and define client connection pool configuration parameters based on the cloud configuration file; where the client connection pool configuration parameters include the number of client connections, client connection configuration information, and connection pool parameters.

[0012] Build the client connection pool in the cloud based on the cloud configuration file.

[0013] In the above solution, by creating a cloud configuration file, the various parameters of the client connection pool can be centrally managed and configured, avoiding the complexity of decentralized configuration and simplifying the management of the system. By defining the number of client connections, the number of concurrent connections can be controlled, avoiding resource exhaustion caused by too many connections and also avoiding performance bottlenecks caused by insufficient connections. The connection pool can effectively manage client connections, ensuring that when some connections fail, connections can be quickly reallocated, improving the high availability and fault tolerance of the system.

[0014] In one implementation, each of the shared subscription groups corresponds to receiving vehicle data of a topic, specifically including:

[0015] Define the subscription topic of each of the shared subscription groups, and modify the subscription topics of the clients in each of the shared subscription groups to shared subscriptions;

[0016] Based on the shared subscription control, the clients receiving vehicle data are controlled to share the received vehicle data with the remaining clients within the shared subscription group.

[0017] In the above solution, the clients within each shared subscription group can share the received vehicle data, ensuring that the clients within the same group can all receive the same data, achieving redundant storage and backup of the data. Even if a certain client fails, other clients can still receive and process the same data, enhancing the fault tolerance of the system. Through the shared subscription mechanism, the vehicle data can be randomly distributed to any client within the shared subscription group, achieving load balancing and avoiding the situation of a single client being overloaded.

[0018] In one implementation, the control server calls the corresponding shared subscription group according to the topic type of the received vehicle data, and randomly sends the vehicle data to any one client within the shared subscription group, specifically including:

[0019] Pre-define the communication protocol between the vehicle side and the server, and parse the topic type of the vehicle data based on the communication protocol;

[0020] Obtain the subscription topics of each of the shared subscription groups, perform a consistency match between the topic type and the subscription topics, and call the first shared subscription group consistent with the topic type according to the match result;

[0021] Randomly select a client within the first shared subscription group based on a random function, and send the vehicle data to the selected client.

[0022] In the above solution, by pre-defining the communication protocol between the vehicle side and the server, the topic type of the vehicle data can be efficiently parsed, ensuring the rapid identification and classification processing of the data. Perform a consistency match between the topic type of the vehicle data and the subscription topics of the shared subscription groups to ensure that the data is accurately distributed to the corresponding shared subscription groups, avoiding incorrect data distribution. Randomly select a client within the shared subscription group based on a random function, achieving random distribution of the data, ensuring load balancing of the clients within the group, and avoiding the situation of a single client being overloaded.

[0023] In one implementation, the cloud data processing method further includes updating the configuration parameters of the thread pool according to the real-time resource utilization rate, specifically:

[0024] Calculate the real-time values of the configuration parameters of the thread pool; wherein, the configuration parameters include the number of core threads, the maximum number of threads, and the queue capacity;

[0025] Generate a dynamic adjustment factor according to the real-time resource utilization rate; wherein, the expression of the dynamic adjustment factor is:

[0026]

[0027] In the formula, δ(t) is the dynamic adjustment factor; Cu(t) is the current CPU utilization rate; Mu(t) is the current memory utilization rate;

[0028] Update the configuration parameters based on the dynamic adjustment factor to obtain updated values of the configuration parameters; wherein, the expression of the updated values of the configuration parameters is:

[0029] N c (t) = N c ×(1 + δ(t));

[0030] N m (t) = N m ×(1 + δ(t));

[0031] Q(t) = Q × (1 + δ(t));

[0032] In the formula, N c (t) is the updated value of the number of core threads; N c is the real-time value of the number of core threads; δ(t) is the dynamic adjustment factor; N m (t) is the updated value of the maximum number of threads; N m is the real-time value of the maximum number of threads; Q(t) is the updated value of the queue capacity; Q is the real-time value of the queue capacity.

[0033] In the above solution, by dynamically adjusting the number of core threads, the maximum number of threads, and the queue capacity of the thread pool according to the real-time CPU utilization rate and memory utilization rate, the system can better adapt to different load conditions, ensure sufficient resources to process data under high load, and save resources under low load.

[0034] In a second aspect, the present application further provides a cloud data processing device, including: a client processing module, a data sending module, and a data processing module;

[0035] The client processing module is used to build a client connection pool in the cloud, connect a number of clients based on the client connection pool, and divide the clients into a number of shared subscription groups; wherein, each shared subscription group corresponds to receiving vehicle data of a topic;

[0036] The data sending module is used to control the server to call the corresponding shared subscription group according to the topic type of the received vehicle data, and randomly send the vehicle data to any one of the clients in the shared subscription group; wherein, the server is used to receive the vehicle data uploaded by the vehicle terminal;

[0037] The data processing module is used to send the vehicle data received by the client to a pre-constructed thread pool for asynchronous processing.

[0038] In the above solution, by constructing a client connection pool, the client connections can be effectively managed and allocated, avoiding overloading of a single client. Dividing the clients into multiple shared subscription groups, with each group corresponding to the vehicle data of a topic, can achieve load balancing and ensure that the clients within each shared subscription group can share the processing tasks. The clients within each shared subscription group can back up each other. Even if a certain client fails, other clients can still continue to process the data, improving the high availability of the system. Sending the vehicle data received by the client to a pre-constructed thread pool for asynchronous processing improves the efficiency and response speed of data processing, and avoids the blocking and delay that may be caused by synchronous processing. The thread pool can effectively manage multi-threaded tasks, optimize resource utilization, and improve the processing capacity of the system. Through the combination of the shared subscription group and the client connection pool, even if a certain client fails, the system can still continue to run, ensuring the continuity of data processing.

[0039] In one implementation, the client processing module is used to construct a client connection pool in the cloud, specifically including:

[0040] Create a cloud configuration file, and define the client connection pool configuration parameters based on the cloud configuration file; wherein, the client connection pool configuration parameters include the number of client connections, client connection configuration information, and connection pool parameters;

[0041] Construct the client connection pool in the cloud based on the cloud configuration file.

[0042] In one implementation, each of the shared subscription groups corresponds to receiving the vehicle data of a topic, specifically including:

[0043] Define the subscription topic of each shared subscription group, and modify the subscription topics of the clients in each shared subscription group to shared subscriptions;

[0044] Based on the shared subscription, control the clients receiving the vehicle data to share the received vehicle data with the remaining clients in the shared subscription group.

[0045] In one implementation, the data sending module is used to control the server to call the corresponding shared subscription group according to the topic type of the received vehicle data, and randomly send the vehicle data to any one of the clients in the shared subscription group, specifically including:

[0046] Pre-define the communication protocol between the vehicle terminal and the server, and parse the topic type of the vehicle data based on the communication protocol;

[0047] Obtain the subscription topics of each shared subscription group, perform consistency matching between the topic type and the subscription topics, and call the first shared subscription group consistent with the topic type according to the matching result;

[0048] Randomly select a client within the first shared subscription group based on a random function, and send the vehicle data to the selected client.

[0049] In one implementation, the cloud data processing device further includes updating the configuration parameters of the thread pool according to the real-time resource utilization rate, specifically:

[0050] Calculate the real-time value of the configuration parameters of the thread pool; wherein, the configuration parameters include the core thread number, the maximum thread number, and the queue capacity;

[0051] Generate a dynamic adjustment factor according to the real-time resource utilization rate; wherein, the expression of the dynamic adjustment factor is:

[0052]

[0053] In the formula, δ(t) is the dynamic adjustment factor; Cu(t) is the current CPU utilization rate; Mu(t) is the current memory utilization rate;

[0054] Update the configuration parameters based on the dynamic adjustment factor to obtain the updated value of the configuration parameters; wherein, the expression of the updated value of the configuration parameters is:

[0055] N c (t) = N c ×(1 + δ(t));

[0056] N m (t) = N m ×(1 + δ(t));

[0057] Q(t) = Q × (1 + δ(t));

[0058] In the formula, N c (t) is the updated value of the core thread number; N c The real-time value of the core thread number; δ(t) is the dynamic adjustment factor; N m (t) is the updated value of the maximum thread number; Nm is the real-time value of the maximum number of threads; Q(t) is the updated value of the queue capacity; Q is the real-time value of the queue capacity.

[0059] In a third aspect, the present application further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the cloud data processing method described above is implemented.

[0060] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program. Wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the cloud data processing method described above. Description of the Drawings

[0061] Figure 1 is a schematic flowchart of a cloud data processing method provided in an embodiment of the present invention;

[0062] Figure 2 is a schematic block diagram of a cloud data processing device provided in an embodiment of the present invention. Detailed Embodiments

[0063] The following combines the drawings and embodiments to further describe in detail the specific embodiments of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0064] The terms "first" and "second" etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0065] Referring to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0066] Embodiment 1

[0067] See Figure 1 , Figure 1The flowchart shows a method for processing cloud data provided in an embodiment of the present invention. The embodiment of the present invention provides a method for processing cloud data, including steps 101 to 103, and the specific steps are as follows:

[0068] Step 101: Build a client connection pool in the cloud, connect a number of clients based on the client connection pool, and divide the clients into several shared subscription groups; where each of the shared subscription groups corresponds to receiving vehicle data of a topic.

[0069] In one embodiment, the building of the client connection pool in the cloud specifically includes: creating a cloud configuration file, and defining client connection pool configuration parameters based on the cloud configuration file; where the client connection pool configuration parameters include the number of client connections, client connection configuration information, and connection pool parameters; building the client connection pool in the cloud based on the cloud configuration file.

[0070] In the embodiment of the present invention, a cloud configuration file is created to define the configuration parameters of the client connection pool. The configuration file can be stored in the form of key-value pairs to ensure the clarity and easy management of the configuration parameters. The configuration file contains the number of client connections, client connection configuration information, and connection pool parameters. Among them, the number of client connections (client_count) is used to define the total number of clients in the pool; the client connection configuration information (client_config) includes the connection protocol, host address, port number, authentication information, etc.; the connection pool parameters (pool_config) include the maximum number of connections, minimum number of connections, connection timeout, idle connection recovery time, etc. of the connection pool. Then, according to the parameters in the created configuration file, a client connection pool is built in the cloud. The building process of the connection pool generally includes initializing the connection pool, creating client connections, and managing the connections. That is, according to the maximum and minimum number of connections in the configuration file, the connection pool is initialized; according to the client connection configuration information in the configuration file, client connections are created and initialized; the connection timeout and idle connection recovery time are set to ensure the efficient management of the connection pool.

[0071] In one embodiment, each of the shared subscription groups corresponding to receiving vehicle data of a topic specifically includes: defining the subscription topic of each of the shared subscription groups, and modifying the subscription topics of the clients in each of the shared subscription groups to shared subscriptions; controlling the clients receiving vehicle data at the receiving end to share the received vehicle data with the remaining clients in the shared subscription group based on the shared subscription.

[0072] In the embodiments of the present invention, subscription topics for each shared subscription group are defined in a configuration file to ensure that each group subscribes to different topics. This can avoid resource waste and duplicate data processing caused by multiple groups subscribing to the same topic. The clients in the client connection pool are divided into multiple shared subscription groups, and the clients within each group subscribe to the corresponding shared topics. The clients within each group use the shared subscription function to ensure that each message is received by only one client within the group.

[0073] Step 102: The control server calls the corresponding shared subscription group according to the topic type of the received vehicle data and randomly sends the vehicle data to any one client within the shared subscription group; wherein, the server is used to receive the vehicle data uploaded by the vehicle terminal.

[0074] In one embodiment, the control server calls the corresponding shared subscription group according to the topic type of the received vehicle data and randomly sends the vehicle data to any one client within the shared subscription group, which specifically includes: pre-defining the communication protocol between the vehicle terminal and the server, parsing the topic type of the vehicle data based on the communication protocol; obtaining the subscription topics of each shared subscription group, performing consistency matching between the topic type and the subscription topics, and calling the first shared subscription group consistent with the topic type according to the matching result; randomly selecting a client within the first shared subscription group based on a random function and sending the vehicle data to the selected client.

[0075] In the embodiments of the present invention, a communication protocol between the vehicle terminal and the server is established in advance, and based on the communication protocol, the topic format, authentication mechanism, data encryption and decryption methods, etc. for transmitting data between the vehicle terminal and the server are defined. When the vehicle terminal sends vehicle data to the server, it is decrypted based on the data decryption method in the communication protocol, and then through the defined topic format, the topic type of the vehicle data can be quickly parsed using a prefix tree or a hash table. Then, the subscription topics of each shared subscription group are obtained, and the topic type of the vehicle data is matched with the subscription topics to ensure the accuracy of the matching. When randomly selecting a client, the client randomly obtained using the default random function of java can be directly used, or a weighted random algorithm can be considered, and weighted according to the processing capacity and current load of the client. If the selected client fails, another client is automatically selected for data sending, and the failure information is recorded for subsequent analysis. Further, when the client does not confirm receiving the message, a message retry mechanism is supported to ensure reliable message transmission. Exemplarily, in the embodiments of the present invention, the MQTT communication protocol is adopted between the vehicle terminal and the server, and 3 topic formats included in the vehicle data are pre-defined:

[0076] Telemetry data: vehicle / <vehicle_id> / telemetry;

[0077] Location data: vehicle / <vehicle_id> / location;

[0078] Status data: vehicle / <vehicle_id> / status;

[0079] The vehicle terminal sends data to the server via the MQTT protocol. For example, vehicle / 123 / telemetry represents the telemetry data of vehicle 123. After receiving the data, the server parses the topic vehicle / 123 / telemetry and extracts the vehicle_id and data_type. Define the clients connected to the client connection pool and define multiple shared subscription groups in the cloud, such as telemetry_group, location_group, status_group.

[0080] The subscription topics for each group are as follows:

[0081] telemetry_group: vehicle / + / telemetry;

[0082] location_group: vehicle / + / location;

[0083] status_group: vehicle / + / status;

[0084] The server matches the parsed data type telemetry with the subscription topics of the shared subscription groups to find the matching telemetry_group. The server obtains the list of all subscribed clients from the telemetry_group. Based on the processing capacity and current load of the clients, a client is selected using a random algorithm, and the vehicle data is sent to the selected client. Preferably, after receiving the data, the client sends an acknowledgment message. If the server does not receive the acknowledgment message within the specified time, mark the client as faulty and record the fault information. Then select another client and resend the message until the message is confirmed to be received.

[0085] It should be noted that in the embodiments of the present invention, the client refers to the client that establishes a communication protocol with the server. If the communication protocol is the MQTT protocol, the client is an MQTT client, which can be in-vehicle devices of a vehicle, mobile applications, etc. The client is responsible for establishing a connection with the server and sending and receiving messages. The vehicle side refers to the devices installed on the vehicle, and these devices usually communicate with the MQTT broker in the cloud as MQTT clients. The vehicle side devices may need to process instructions from the cloud or send the status data of the vehicle to the cloud. The cloud refers to the cloud platform running the MQTT broker. The MQTT broker is responsible for managing all client connections, forwarding messages, and ensuring that messages are correctly delivered from the sender to the receiver; the server serves as a transfer station for vehicle data.

[0086] Step 103: Send the vehicle data received by the client to a pre-constructed thread pool for asynchronous processing.

[0087] In the embodiments of the present invention, ThreadPoolTaskExecutor is used to construct and manage the thread pool, and appropriate configuration parameters such as the number of threads and queue capacity are configured. ThreadPoolTaskExecutor is a thread pool task executor provided in the Spring framework for implementing asynchronous task processing. Using ThreadPoolTaskExecutor to construct and manage the thread pool is a conventional technical means in the art and will not be limited herein. Preferably, ThreadPoolTaskExecutor can also be injected into the MultiMqttMessageHandler class. ThreadPoolTaskExecutor is a thread pool task executor for managing the execution of multiple threads. Injecting it means using the thread pool to process messages in the MultiMqttMessageHandler class, thereby improving the concurrent processing ability and system response speed.

[0088] In one embodiment, the cloud data processing method further includes updating the configuration parameters of the thread pool according to the real-time resource utilization rate, specifically: calculating the real-time values of the configuration parameters of the thread pool; wherein, the configuration parameters include the core number of threads, the maximum number of threads, and the queue capacity; generating a dynamic adjustment factor according to the real-time resource utilization rate; wherein, the expression of the dynamic adjustment factor is:

[0089]

[0090] In the formula, δ(t) is the dynamic adjustment factor; Cu(t) is the current CPU utilization rate; Mu(t) is the current memory utilization rate;

[0091] Update the configuration parameter based on the dynamic adjustment factor to obtain an updated value of the configuration parameter; wherein, the expression of the updated value of the configuration parameter is:

[0092] N c (t) = N c × (1 + δ(t));

[0093] N m (t) = N m × (1 + δ(t));

[0094] Q(t) = Q × (1 + δ(t));

[0095] In the formula, N c (t) is the updated value of the number of core threads; N c is the real-time value of the number of core threads; δ(t) is the dynamic adjustment factor; N m (t) is the updated value of the maximum number of threads; N m is the real-time value of the maximum number of threads; Q(t) is the updated value of the queue capacity; Q is the real-time value of the queue capacity.

[0096] In the embodiments of the present invention, the CPU and memory utilization rates of the real-time monitoring system are monitored. By monitoring the resource utilization rate in real time, the load situation of the system can be understood in a timely manner, providing a basis for subsequent dynamic adjustment. Exemplarily, monitoring tools of the operating system or third-party libraries (such as Java's ManagementFactory) can be used to obtain the real-time CPU and memory utilization rates. Calculate the real-time values of the number of core threads, the maximum number of threads, and the queue capacity through the configuration parameter calculation formula, where the configuration parameter calculation formula is:

[0097] Nc = [C × α × λ]

[0098] Nm = [Nc × β];

[0099] Q = [Nm × γ]

[0100] Wherein, Nc is the number of core threads; C is the number of CPU cores; α is the I / O waiting ratio factor; λ is the queue depth waiting factor; Nm is the maximum number of threads; β is the memory impact factor; Q is the queue capacity; γ is the queue depth factor;

[0101]

[0102] λ = L × (1 + log2(C));

[0103] In the formula, W is the average I / O waiting time; P is the average CPU processing time; T is the target response time; M is the total system memory; L is the load factor; C is the number of CPU cores.

[0104] Then, a dynamic adjustment factor is generated based on the two resource utilization rates, and the real-time values of the configuration parameters of the thread pool are updated and adjusted accordingly. Exemplarily, if the working data of the thread pool is as follows:

[0105] C = 8 / / 8-core CPU

[0106] M = 16 / / 16GB of memory

[0107] W = 1.5 / / 1.5ms I / O wait

[0108] P = 0.2 / / 0.2ms CPU processing

[0109] L = 0.8 / / Load factor

[0110] T = 200 / / Target response time of 200ms

[0111] Then the calculation process of the real-time values of the configuration parameters is as follows:

[0112]

[0113] λ = L × (1 + log2(C)) = 0.8 × (1 + log2(8)) = 0.8 × (1 + 3) = 3.2;

[0114]

[0115] Then, a dynamic adjustment factor is calculated based on the real-time monitored resource utilization rates, and the real-time values of each configuration parameter are updated according to the calculated dynamic adjustment factor. By dynamically adjusting the core thread number, maximum thread number, and queue capacity, the system can better adapt to the current load condition of the system and improve the processing ability and response speed of the system.

[0116] In an embodiment of the present invention, a cloud data processing device is further provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the above-mentioned cloud data processing method is implemented.

[0117] In an embodiment of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned cloud data processing method.

[0118] Exemplarily, the computer program can be divided into one or more modules. One or more modules are stored in the memory and executed by the processor to complete the present invention. One or more modules can be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the cloud data processing device.

[0119] The cloud data processing device may be a computing device such as a desktop computer, a notebook, a handheld computer, and a cloud server. The cloud data processing device may include, but is not limited to, a processor, a memory, and a display. Those skilled in the art can understand that the above components are only examples of the cloud data processing device and do not constitute a limitation on the cloud data processing device. It may include more or fewer components than those described, or combine certain components, or have different components. For example, the cloud data processing device may also include input / output devices, network access devices, a bus, etc.

[0120] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the cloud data processing device and connects various parts of the entire cloud data processing device through various interfaces and lines.

[0121] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the cloud data processing device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, a text conversion function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0122] Among them, when the module for processing cloud data is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0123] An embodiment of the present invention provides a cloud data processing method. By constructing a client connection pool, client connections can be effectively managed and allocated, avoiding overloading of a single client. The clients are divided into multiple shared subscription groups, and each group corresponds to vehicle data of a topic, which can achieve load balancing and ensure that the clients within each shared subscription group can share the processing tasks. The clients within each shared subscription group can back up each other. Even if a certain client fails, other clients can still continue to process data, improving the high availability of the system. The vehicle data received by the clients is sent to a pre-constructed thread pool for asynchronous processing, improving the efficiency and response speed of data processing and avoiding the blocking and delay that may be caused by synchronous processing. The thread pool can effectively manage multi-threaded tasks, optimize resource utilization, and improve the processing capacity of the system. Through the combination of the shared subscription group and the client connection pool, even if a certain client fails, the system can still continue to run, ensuring the continuity of data processing.

[0124] Embodiment 2

[0125] See Figure 2 , Figure 2 is a schematic diagram of the modules of a cloud data processing device provided in an embodiment of the present invention. An embodiment of the present invention provides a cloud data processing device, including: a client processing module 201, a data sending module 202, and a data processing module 203;

[0126] The client processing module 201 is used to build a client connection pool in the cloud, connect several clients based on the client connection pool, and divide the clients into several shared subscription groups; wherein, each of the shared subscription groups corresponds to receiving vehicle data of a topic.

[0127] The data sending module 202 is used to control the server to call the corresponding shared subscription group according to the topic type of the received vehicle data, and randomly send the vehicle data to any one client within the shared subscription group; wherein, the server is used to receive the vehicle data uploaded by the vehicle terminal.

[0128] The data processing module 203 is used to send the vehicle data received by the client to a pre-built thread pool for asynchronous processing.

[0129] In one embodiment, the client processing module 201 is used to build a client connection pool in the cloud, specifically including: creating a cloud configuration file, and defining client connection pool configuration parameters based on the cloud configuration file; wherein, the client connection pool configuration parameters include the number of client connections, client connection configuration information, and connection pool parameters; building the client connection pool in the cloud based on the cloud configuration file.

[0130] In one embodiment, each of the shared subscription groups corresponds to receiving vehicle data of a topic, specifically including: defining the subscription topic of each shared subscription group, and modifying the subscription topic of the clients in each shared subscription group to shared subscription; controlling the clients receiving the vehicle data to share the received vehicle data with the remaining clients within the shared subscription group based on the shared subscription.

[0131] In one embodiment, the data sending module 202 is used to control the server to call the corresponding shared subscription group according to the topic type of the received vehicle data, and randomly send the vehicle data to any one client within the shared subscription group, specifically including: pre-defining the communication protocol between the vehicle terminal and the server, parsing the topic type of the vehicle data based on the communication protocol; obtaining the subscription topic of each shared subscription group, performing consistency matching between the topic type and the subscription topic, and calling the first shared subscription group consistent with the topic type according to the matching result; randomly selecting a client within the first shared subscription group based on a random function, and sending the vehicle data to the selected client.

[0132] In one embodiment, the cloud data processing device further includes updating the configuration parameters of the thread pool according to the real-time resource utilization rate, specifically: calculating the real-time value of the configuration parameters of the thread pool; wherein, the configuration parameters include the number of core threads, the maximum number of threads, and the queue capacity.

[0133] Generate a dynamic adjustment factor according to the real-time resource utilization rate; wherein, the expression of the dynamic adjustment factor is:

[0134]

[0135] In the formula, δ(t) is the dynamic adjustment factor; Cu(t) is the current CPU utilization rate; Mu(t) is the current memory utilization rate;

[0136] Update the configuration parameters based on the dynamic adjustment factor to obtain an updated value of the configuration parameters; wherein, the expression of the updated value of the configuration parameters is:

[0137] N c (t) = N c ×(1 + δ(t));

[0138] N m (t) = N m ×(1 + δ(t));

[0139] Q(t) = Q × (1 + δ(t));

[0140] In the formula, N c (t) is the updated value of the core thread count; N c is the real-time value of the core thread count; δ(t) is the dynamic adjustment factor; N m (t) is the updated value of the maximum thread count; N m is the real-time value of the maximum thread count; Q(t) is the updated value of the queue capacity; Q is the real-time value of the queue capacity.

[0141] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described device can refer to the corresponding process in the foregoing method embodiment, and will not be elaborated herein.

[0142] The embodiment of the present invention provides a cloud data processing device. By constructing a client connection pool, it can effectively manage and allocate client connections, avoiding overloading of a single client. The clients are divided into multiple shared subscription groups, and each group corresponds to vehicle data of a topic, which can achieve load balancing and ensure that the clients within each shared subscription group can share the processing tasks. The clients within each shared subscription group can back up each other. Even if a certain client fails, other clients can still continue to process data, improving the high availability of the system. The vehicle data received by the clients is sent to a pre-constructed thread pool for asynchronous processing, improving the efficiency and response speed of data processing and avoiding the blocking and delay that may be caused by synchronous processing. The thread pool can effectively manage multi-threaded tasks, optimize resource utilization, and improve the processing capacity of the system. Through the combination of the shared subscription group and the client connection pool, even if a certain client fails, the system can still continue to run, ensuring the continuity of data processing.

[0143] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and substitutions can be made, and these improvements and substitutions should also be regarded as the protection scope of the present invention.

Claims

1. A cloud data processing method, characterized in that: include: Building a client connection pool in the cloud, connecting a number of clients based on the client connection pool and dividing the clients into a number of shared subscription groups; wherein each of the shared subscription groups receives vehicle data of a topic; The control server calls the corresponding shared subscription group according to the subject type of the received vehicle data, and randomly sends the vehicle data to any client in the shared subscription group; wherein the server is used to receive the vehicle data uploaded by the vehicle end; The vehicle data received by the client is sent to a pre-built thread pool for asynchronous processing.

2. A cloud data processing method according to claim 1, characterized in that: The construction of the client connection pool in the cloud specifically includes: Creating a cloud configuration file, and defining client connection pool configuration parameters based on the cloud configuration file; wherein the client connection pool configuration parameters include the number of client connections, client connection configuration information, and connection pool parameters; The client connection pool is constructed in the cloud based on the cloud configuration file.

3. A cloud data processing method according to claim 1, characterized in that: Each of the shared subscription groups receives vehicle data of a topic, specifically including: Defining a subscription topic for each of the shared subscription groups, and modifying the subscription topic of the client in each of the shared subscription groups to a shared subscription; Based on the shared subscription control, the client receiving the vehicle data shares the received vehicle data with other clients in the shared subscription group.

4. A cloud data processing method according to claim 1, characterized in that: The control server calls the corresponding shared subscription group according to the subject type of the received vehicle data, and randomly sends the vehicle data to any client in the shared subscription group, specifically including: pre-define a communication protocol between the vehicle end and the server, and parse the subject type of the vehicle data based on the communication protocol; Acquire a subscription topic of each of the shared subscription groups, match the topic type with the subscription topic for consistency, and call a first shared subscription group consistent with the topic type according to the matching result; A client is randomly selected in the first shared subscription group based on a random function, and the vehicle data is sent to the selected client.

5. A cloud data processing method according to claim 1, characterized in that: The cloud data processing method further includes updating the configuration parameters of the thread pool according to the real-time resource utilization, specifically: Calculating real-time values ​​of configuration parameters of the thread pool; wherein the configuration parameters include the number of core threads, the maximum number of threads, and the queue capacity; Generate a dynamic adjustment factor according to the real-time resource utilization; wherein the expression of the dynamic adjustment factor is: Where δ(t) is the dynamic adjustment factor; Cu(t) is the current CPU utilization; Mu(t) is the current memory utilization; The configuration parameter is updated based on the dynamic adjustment factor to obtain an updated value of the configuration parameter; wherein the expression of the updated value of the configuration parameter is: N c (t)=N c ×(1+δ(t)); N m (t)=N m ×(1+δ(t)); Q(t) = Q × (1 + δ(t)); Where N c (t) is the updated value of the number of core threads; N c The real-time value of the number of core threads; δ(t) is the dynamic adjustment factor; N m (t) is the updated value of the maximum number of threads; N m is the real-time value of the maximum number of threads; Q(t) is the updated value of the queue capacity; Q is the real-time value of the queue capacity.

6. A cloud data processing device, characterized in that: include: Client processing module, data sending module and data processing module; The client processing module is used to build a client connection pool in the cloud, connect a number of clients based on the client connection pool and divide the clients into a number of shared subscription groups; wherein each shared subscription group receives vehicle data of a topic correspondingly; The data sending module is used to control the server to call the corresponding shared subscription group according to the subject type of the received vehicle data, and randomly send the vehicle data to any client in the shared subscription group; wherein the server is used to receive the vehicle data uploaded by the vehicle end; The data processing module is used to send the vehicle data received by the client to a pre-built thread pool for asynchronous processing.

7. A cloud data processing device as claimed in claim 6, characterized in that: The client processing module is used to build a client connection pool in the cloud, specifically including: Creating a cloud configuration file, and defining client connection pool configuration parameters based on the cloud configuration file; wherein the client connection pool configuration parameters include the number of client connections, client connection configuration information, and connection pool parameters; The client connection pool is constructed in the cloud based on the cloud configuration file.

8. A cloud data processing device as claimed in claim 6, characterized in that: Each of the shared subscription groups receives vehicle data of a topic, specifically including: Defining a subscription topic for each of the shared subscription groups, and modifying the subscription topic of the client in each of the shared subscription groups to a shared subscription; Based on the shared subscription control, the client receiving the vehicle data shares the received vehicle data with other clients in the shared subscription group.

9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the cloud data processing method as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the cloud data processing method as described in any one of claims 1 to 5.

Images