Data processing method and device and related equipment

By loading and updating status data in the server's local database, the problem of low data processing efficiency in cloud computing is solved, and the effect of quickly obtaining the latest status data and improving computing efficiency is achieved.

CN118939695APending Publication Date: 2024-11-12CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410987304.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The data processing efficiency in the existing cloud computing field is low, mainly because the intermediate data generated during the calculation process cannot be stored locally, resulting in the need to re-acquire the data when processing errors occur, resulting in inefficiency.

Method used

After loading the first status data in the server's local database for processing, and generating the target calculation results, update the data in the local database and cloud server so that the latest status data will be quickly obtained during the next calculation.

Benefits of technology

By processing and updating data locally, the efficiency of data processing is improved, the need for reprocessing is avoided, and the computing efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939695A_ABST
    Figure CN118939695A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device and related equipment, and is applied to a server, and the method comprises the steps: receiving a data processing request sent by a client; loading first state data in a local database of the server according to the data processing request; processing the to-be-processed data based on the first state data to obtain a target calculation result; and sending the target calculation result to the client, updating the first state data in the local database based on the target calculation result, and sending the target calculation result to a cloud server. In the data processing process, the first state data is loaded in the local database of the server through the data processing request, the to-be-processed data is processed through the first state data, the target calculation result is generated, and the local database and the cloud server are updated, so that the data processing efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of cloud computing, and in particular, to a data processing method, apparatus, and related equipment. Background Art

[0002] In recent years, cloud computing has been widely used in many fields. In the field of cloud computing, the processing efficiency of source data can be improved by stream processing of source data. In existing processing schemes, since the intermediate data generated during the calculation process cannot be stored locally, when errors occur in the processing of source data, it is necessary to obtain the data again in the cloud for processing, which leads to the problem of low data processing efficiency. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, device and related equipment to solve the problem of low data processing efficiency in the prior art.

[0004] To solve the above problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a server, and the method includes:

[0006] Receiving a data processing request sent by a client, the data processing request including data to be processed, and the data processing request is used to request the server to process the data to be processed;

[0007] Loading first state data in a local database of the server according to the data processing request, wherein the first state data is used to process the data to be processed;

[0008] Processing the data to be processed based on the first state data to obtain a target calculation result;

[0009] The target calculation result is sent to the client, and the first state data in the local database is updated based on the target calculation result. The target calculation result is sent to the cloud server so that the cloud server updates the stored first state data based on the target calculation result.

[0010] Optionally, the server includes a first computing unit and a second computing unit, and the processing of the to-be-processed data based on the first state data to obtain a target computing result includes:

[0011] The first computing unit processes the data to be processed using the first state data to obtain a first computing result;

[0012] Storing the first calculation result in the local database;

[0013] The second calculation unit processes the data to be processed using the first calculation result to obtain a second calculation result;

[0014] The first calculation result and the second calculation result are combined to obtain the target calculation result.

[0015] Optionally, loading the first state data in a local database of the server according to the data processing request includes:

[0016] When the first state data is loaded into the local database according to the data processing request and the loading fails, the first state data is loaded into the cloud server according to the data processing request.

[0017] Optionally, the data to be processed includes N source data, where N is an integer greater than 1. After receiving the data processing request sent by the client, the method further includes:

[0018] The N source data are merged based on a preset threshold to obtain at least one merged data, wherein the data to be processed includes the at least one merged data, and the number of source data included in the merged data is less than or equal to the preset threshold.

[0019] Optionally, the server is provided with a first logical queue and a second logical queue, and the method further includes:

[0020] receiving a capacity expansion request, the capacity expansion request being used to expand the storage capacity of the first logical queue from a first storage capacity to a second storage capacity, the first logical queue comprising at least two first physical queues;

[0021] generating a second physical queue in the server according to the capacity expansion request, wherein the storage capacity of the second physical queue is a target storage capacity, and the target storage capacity is a difference between the second storage capacity and the first storage capacity;

[0022] The second physical queue is added to the second logical queue, and the second physical queue is serially connected to the last first physical queue of the at least two first physical queues.

[0023] Optionally, the server includes a first logical queue and a second logical queue, the first logical queue includes at least two first physical queues, the second logical queue includes a third physical queue, and the third physical queue is connected in series with the last first physical queue of the at least two first physical queues, and the method further includes:

[0024] Receive a reduction request;

[0025] In response to the shrink request, the third physical queue is unbound in the second logical queue.

[0026] In a second aspect, an embodiment of the present application further provides a data processing device, applied to a server, the device comprising:

[0027] A receiving module, used for receiving a data processing request sent by a client, wherein the data processing request includes data to be processed, and the data processing request is used for requesting the server to process the data to be processed;

[0028] A loading module, used for loading first state data in a local database of the server according to the data processing request, wherein the first state data is used for processing the data to be processed;

[0029] A processing module, used for processing the data to be processed based on the first state data to obtain a target calculation result;

[0030] A sending module is used to send the target calculation result to the client, update the first state data in the local database based on the target calculation result, and send the target calculation result to the cloud server so that the cloud server updates the stored first state data based on the target calculation result.

[0031] In a third aspect, an embodiment of the present application further provides a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the method described in the first aspect above.

[0032] In a fourth aspect, an embodiment of the present application further provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the method described in the first aspect above.

[0033] In a fifth aspect, an embodiment of the present application further provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps in the method described in the first aspect.

[0034] The present application provides a data processing method, device and related equipment, which are applied to a server, and the method includes: receiving a data processing request sent by a client, the data processing request includes data to be processed, and the data processing request is used to request the server to process the data to be processed; loading first state data in the local database of the server according to the data processing request, the first state data is used to process the data to be processed; processing the data to be processed based on the first state data to obtain a target calculation result; sending the target calculation result to the client, and updating the first state data in the local database based on the target calculation result, and sending the target calculation result to a cloud server, so that the cloud server updates the stored first state data based on the target calculation result. In the data processing process, the present application loads the first state data in the local database of the server through the data processing request, thereby processing the data to be processed through the first state data to generate the target calculation result, and thereby updating the local database and the cloud server, thereby improving the data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 It is a flowchart of a method for processing data provided in an embodiment of the present application;

[0037] Figure 2 It is an interactive schematic diagram provided in an embodiment of the present application;

[0038] Figure 3 It is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0039] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] The terms "first", "second" etc. in the embodiments of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of connected objects, such as A and / or B and / or C, representing the inclusion of separate A, separate B, separate C, and A and B all exist, B and C all exist, A and C all exist, and 7 situations that A, B and C all exist.

[0042] See also Figure 1 , Figure 1 It is a flowchart of the data processing method provided in the embodiment of the present application. Figure 1 The data processing method shown is applied to the server, such as Figure 1 As shown, the data processing method may include the following steps:

[0043] Step 101: Receive a data processing request sent by a client, wherein the data processing request includes data to be processed, and the data processing request is used to request the server to process the data to be processed.

[0044] In this embodiment, the server is illustrated by taking the open source RocketMQ Streams architecture as an example, wherein RocketMQ is a distributed message middleware with low latency, high concurrency, high availability and high reliability. RocketMQ can provide asynchronous decoupling and peak-shaving capabilities for distributed application systems, and also has the characteristics of massive message accumulation, high throughput, reliable retry and other features required by Internet applications. Since its birth, RocketMQ has been widely adopted by many enterprise developers and cloud vendors due to its simple architecture, rich business functions and strong scalability. After more than ten years of large-scale scene polishing, RocketMQ has become the industry consensus of the preferred solution for reliable financial-grade business messages, and is widely used in business scenarios in the fields of the Internet, big data, mobile Internet, and the Internet of Things. The open source RocketMQ Streams architecture consumes upstream data through RocketMQ, enters the processing topology and is processed by the computing instance, and after the calculation is completed, the results are written back to RocketMQ.

[0045] In addition, in this application, the data is processed by stream processing, where stream processing refers to the technology of real-time processing of data streams, and data streams refer to a series of continuously generated data records, which can be from various data sources such as sensors, server logs, and network data. The stream processing framework can process and analyze this data in real time and extract valuable information to support application scenarios such as real-time decision-making and real-time monitoring. Compared with traditional batch processing, stream processing has faster response speed and higher flexibility, and is suitable for scenarios that require real-time analysis of data and rapid response, such as finance, telecommunications, and the Internet of Things.

[0046] Specifically, the data processing request sent by the client is used to process the data to be processed, wherein the data processing request carries the data to be processed, and the data processing request is used to process the data to be processed.

[0047] Step 102: Load first state data into the local database of the server according to the data processing request, where the first state data is used to process the data to be processed.

[0048] In this embodiment, after receiving a data processing request, first state data is loaded into the local database of the server, wherein the first state data is intermediate data of the data to be processed. The computational efficiency of the data to be processed can be improved through the intermediate data, and when problems occur in the calculation of the data to be processed, the calculation can be restored in time through the intermediate data, thereby avoiding reprocessing.

[0049] Step 103: Process the data to be processed based on the first state data to obtain a target calculation result.

[0050] In this embodiment, the data to be processed is processed according to the first state data, and after the processing is completed, a target calculation result is generated. The target calculation result is used to represent the processing result of the server on the data to be processed. Specifically, the processing process of the data to be processed is not specifically limited in this embodiment. For example, it can be to perform calculations on the data to be processed, or to perform statistics on the data to be processed, etc.

[0051] Step 104: Send the target calculation result to the client, update the first state data in the local database based on the target calculation result, and send the target calculation result to the cloud server so that the cloud server updates the stored first state data based on the target calculation result.

[0052] In this embodiment, after the target calculation result is generated by the server, the first state data in the local database is updated by the target calculation result, and the target calculation result is sent to the cloud server, so that the cloud server updates the first state data, thereby storing the latest state data. Therefore, in the next calculation of the same data to be processed, the latest state data can be quickly obtained, thereby improving the calculation efficiency.

[0053] The present application provides a data processing method, which is applied to a server, and the method includes: receiving a data processing request sent by a client, the data processing request includes data to be processed, and the data processing request is used to request the server to process the data to be processed; loading first state data in the local database of the server according to the data processing request, the first state data is used to process the data to be processed; processing the data to be processed based on the first state data to obtain a target calculation result; sending the target calculation result to the client, and updating the first state data in the local database based on the target calculation result, and sending the target calculation result to a cloud server, so that the cloud server updates the stored first state data based on the target calculation result. In the process of data processing, the present application loads the first state data in the local database of the server through the data processing request, thereby processing the data to be processed through the first state data, generating a target calculation result, and thereby updating the local database and the cloud server, thereby improving the data processing efficiency.

[0054] In some feasible implementations, optionally, the server includes a first computing unit and a second computing unit, and the processing of the data to be processed based on the first state data to obtain a target computing result includes:

[0055] The first computing unit processes the data to be processed using the first state data to obtain a first computing result;

[0056] Storing the first calculation result in the local database;

[0057] The second calculation unit processes the data to be processed using the first calculation result to obtain a second calculation result;

[0058] The first calculation result and the second calculation result are combined to obtain the target calculation result.

[0059] In this embodiment, if Figure 2 As shown, Figure 2This is a schematic diagram of server processing in this embodiment. Specifically, in stream processing, the Source end is the source of data, there is a computing instance in the middle, and finally the data will enter the Sink end. Usually they are all stateful. For example, to calculate the word count, each piece of data comes in, and a new word appears. First, the count value of the previous data must be taken, and then 1 is added to generate a new count value. The data in the middle is the state data. The most important requirement for the storage of this state data is to ensure high performance while ensuring high availability. In response to these two requirements, this solution proposes a method of using local RocksDB combined with remote RocketMQ to store state data. When the stream processing is in the computing state, RocksDB is used as a temporary storage of state data for interacting with the computing instance. When the consumption site is submitted after the calculation is completed, the state generated by this calculation will be written to RocketMQ.

[0060] Specifically, process1 is the first computing unit, process2 is the second computing unit, and state topic is the cloud server. In this embodiment, the first computing unit uses the first state data to process the data to be processed. After obtaining the first computing result, the first computing result is stored in the local database and the first computing result is sent to the cloud server to update and store the first state data. The second computing unit is different from the first computing unit, and the algorithms therein are different, so the calculation results obtained are also different. It should be noted that in other implementations, there may be multiple computing units, and the algorithms of different computing units are different, which is not specifically limited in this embodiment.

[0061] Optionally, loading the first state data in a local database of the server according to the data processing request includes:

[0062] When the first state data is loaded into the local database according to the data processing request and the loading fails, the first state data is loaded into the cloud server according to the data processing request.

[0063] In this embodiment, for high performance, a localized RocksDB is used to store state data. Putting RocksDB and computing nodes together, fetching data from RocksDB is equivalent to fetching data locally, which improves performance. Among them, for high availability, a new state topic is used in the message queue RocketMQ. The state topic is based on the data expiration mechanism of the Key. For the data with the same Key, only the latest value is retained. For the data reprocessing problem in the stream processing scenario, the statetopic can make the amount of data that needs to be read during recovery very small. And the stored data is saved in the message queue RocketMQ. If a computing node A crashes, when computing node B takes over the task, the state data can be read by remotely consuming the messages in the statetopic, and then stored in RocksDB after local recovery, thus achieving high availability.

[0064] For computing instances of stream processing, it is necessary to efficiently obtain state data and achieve high availability of computing instances and lightweight state replay. We propose a storage architecture that combines local RocksDB with remote RocketMQ. The state topic that stores remote state data is based on the data expiration mechanism of the key, and only retains the latest value for data with the same key. The state data relies on local RocksDB to accelerate reading, and remote RocketMQ for persistence.

[0065] Optionally, the data to be processed includes N source data, where N is an integer greater than 1. After receiving the data processing request sent by the client, the method further includes:

[0066] The N source data are merged based on a preset threshold to obtain at least one merged data, wherein the data to be processed includes the at least one merged data, and the number of source data included in the merged data is less than or equal to the preset threshold.

[0067] In this embodiment, when receiving the data to be processed, multiple source data in the data to be processed may be merged to obtain a merged data, and the merged data may be processed uniformly, thereby improving processing efficiency.

[0068] Specifically, when the client consumes the source data in the Source Topic, a new batch indexing method is used to improve throughput. The main implementation is automatic assembly on the client, merging multiple messages according to topics and queues, and sending them as a request to the server; the server does not decompress the received messages, but directly stores them; the consumer obtains a batch at a time, takes multiple messages to the local, and then decompresses them. Originally, an RPC request was required for each message, but after using batch indexing, an RPC request can be made once for 10 messages.

[0069] On the client side, multiple messages are merged according to topic and queue, and sent to the server side as a request. The server side does not decompress the received messages, but directly stores them. The consumer side takes multiple messages to the local side and then decompresses them. Distributing the decompression operation to multiple consumers reduces the pressure on the server side and improves the efficiency of stream processing.

[0070] Optionally, the server is provided with a first logical queue and a second logical queue, and the method further includes:

[0071] receiving a capacity expansion request, the capacity expansion request being used to expand the storage capacity of the first logical queue from a first storage capacity to a second storage capacity, the first logical queue comprising at least two first physical queues;

[0072] generating a second physical queue in the server according to the capacity expansion request, wherein the storage capacity of the second physical queue is a target storage capacity, and the target storage capacity is a difference between the second storage capacity and the first storage capacity;

[0073] The second physical queue is added to the second logical queue, and the second physical queue is serially connected to the last first physical queue of the at least two first physical queues.

[0074] In this embodiment, in the stream processing scenario, another typical problem is the expansion and contraction of the queue and the rebalancing of data. Before and after the number of queues is expanded or contracted, the same key may be sent to different queues. In order to solve this problem, this design scheme has made a deep transformation to the queue model of RocketMQ, the message queue that acts as the base, and proposed a model of logical queues, sinking the current queue to a physical queue. The upper layer re-abstracted the logical queue, fixed the number of Source Topic and Sink Topic queues in the entire stream processing process, and expanded and contracted the underlying physical queues insensibly. The logical queue is the queue exposed to the customer, and the physical queue is the queue used to actually store data. A logical queue is distributed on different physical storage nodes (Broker) through horizontal sharding and vertical segmentation. Horizontal sharding solves the problem of high availability, and multiple shards of the same logical queue can be randomly written at multiple points. Vertical segmentation solves the problem of logical queue expansion, and through multi-level queue mapping, the underlying physical queue can be expanded or contracted insensibly. The offset in the logical queue is monotonically increasing. The offsets in the physical queue are monotonically increasing and continuous and belong to the queues on the same Broker. The numbers are monotonically increasing and continuous. This ensures that data with the same key can be sent to the same queue. At the same time, the concept of Leader Queue is proposed to store the mapping relationship between logical queues and physical queues. Leader Queue is the latest physical queue mapped to a logical queue, that is, the writable queue. Following the Leader Completeness principle, for each logical queue, all mapping relationships are stored on the Broker where the latest queue is located, avoiding data cutting and facilitating compression and optimization of mapping relationship data.

[0075] When expanding, the second physical queue is generated in the second logical queue, and then the second physical queue is connected in series to the first logical queue, thereby completing the expansion of the first logical queue. Specifically, the offset of the logical queue logic-queue-2 before expansion is 0-1000. When expanding, the physical queue queue-3 where the new node is located is bound to itself and connected in series to the back of the physical queue queue-2 of the original node, forming a new logical queue with an offset of 0-2000.

[0076] Optionally, the server includes a first logical queue and a second logical queue, the first logical queue includes at least two first physical queues, the second logical queue includes a third physical queue, and the third physical queue is connected in series with the last first physical queue of the at least two first physical queues, and the method further includes:

[0077] Receive a reduction request;

[0078] In response to the shrink request, the third physical queue is unbound in the second logical queue.

[0079] In this embodiment, in contrast to the above expansion process, the reduction can be completed by simply unbinding the physical queue queue-3 from the logical queue logic-queue-2. Thus, the server can be expanded without the need to transport data and the shard data does not need to change.

[0080] Directly increasing or decreasing the number of queues when scaling the cluster will cause changes in the number of queues and the distribution of the entire data, and the same key may be sent to different queues. A logical queue model is proposed, which sinks the current queue into a physical queue. The upper layer re-abstracts the logical queue. A logical queue is distributed on different physical storage nodes (Broker) through horizontal sharding and vertical segmentation. Horizontal sharding solves the high availability problem, and vertical segmentation uses multi-level queue mapping to fix the number of queues, thereby achieving seamless scaling of the underlying physical queues and ensuring that data with the same key can be sent to the same queue.

[0081] During the data processing process, the present application loads the first state data into the local database of the server through a data processing request, thereby processing the data to be processed through the first state data to generate a target calculation result, and thereby updating the local database and the cloud server, thereby improving the data processing efficiency.

[0082] See also Figure 3 , Figure 3 is a structural diagram of a data processing device provided in an embodiment of the present application. Figure 3 As shown, the data processing device 300 is applied to the server, and the data processing device 300 includes:

[0083] The receiving module 310 is used to receive a data processing request sent by a client, wherein the data processing request includes data to be processed, and the data processing request is used to request the server to process the data to be processed;

[0084] A loading module 320, configured to load first state data in a local database of the server according to the data processing request, wherein the first state data is used to process the data to be processed;

[0085] A processing module 330 is used to process the data to be processed based on the first state data to obtain a target calculation result;

[0086] The sending module 340 is used to send the target calculation result to the client, update the first state data in the local database based on the target calculation result, and send the target calculation result to the cloud server so that the cloud server updates the stored first state data based on the target calculation result.

[0087] Optionally, the server includes a first computing unit and a second computing unit, and the processing module 330 includes:

[0088] A first processing submodule, configured to process the data to be processed using the first state data based on the first computing unit to obtain a first computing result;

[0089] A storage submodule, used for storing the first calculation result in the local database;

[0090] A second processing submodule, configured to process the data to be processed using the first calculation result based on the second calculation unit to obtain a second calculation result;

[0091] The merging submodule is used to merge the first calculation result and the second calculation result to obtain the target calculation result.

[0092] Optionally, the loading module 320 includes:

[0093] The loading submodule is used to load the first state data in the local database according to the data processing request, and if the loading fails, load the first state data in the cloud server according to the data processing request.

[0094] Optionally, the data to be processed includes N source data, where N is an integer greater than 1, and further includes:

[0095] A merging module is used to merge the N source data based on a preset threshold to obtain at least one merged data, wherein the data to be processed includes the at least one merged data, and the number of source data included in the merged data is less than or equal to the preset threshold.

[0096] Optionally, also include:

[0097] an expansion request receiving module, configured to receive an expansion request, wherein the expansion request is used to expand the storage capacity of the first logical queue from a first storage capacity to a second storage capacity, wherein the first logical queue includes at least two first physical queues;

[0098] an expansion request generating module, configured to generate a second physical queue in the server according to the expansion request, wherein the storage capacity of the second physical queue is a target storage capacity, and the target storage capacity is a difference between the second storage capacity and the first storage capacity;

[0099] An adding module is used to add the second physical queue to the second logical queue, and to connect the second physical queue in series to the last first physical queue of the at least two first physical queues.

[0100] Optionally, the server includes a first logical queue and a second logical queue, the first logical queue includes at least two first physical queues, the second logical queue includes a third physical queue, the third physical queue is connected in series with the last first physical queue of the at least two first physical queues, and further includes:

[0101] A shrink request receiving module, used for receiving a shrink request;

[0102] An unbinding module is used to unbind the third physical queue in the second logical queue in response to the shrink request.

[0103] During the data processing process, the present application loads the first state data into the local database of the server through a data processing request, thereby processing the data to be processed through the first state data to generate a target calculation result, and thereby updating the local database and the cloud server, thereby improving the data processing efficiency.

[0104] The present application also provides an electronic device. Figure 4 , the electronic device may include a processor 401, a memory 402, and a program 4021 stored in the memory 402 and executable on the processor 401.

[0105] When the program 4021 is executed by the processor 401, it can achieve Figure 1 Any step in the corresponding method embodiment:

[0106] Receiving a data processing request sent by a client, the data processing request including data to be processed, and the data processing request is used to request the server to process the data to be processed;

[0107] Loading first state data in a local database of the server according to the data processing request, wherein the first state data is used to process the data to be processed;

[0108] Processing the data to be processed based on the first state data to obtain a target calculation result;

[0109] The target calculation result is sent to the client, and the first state data in the local database is updated based on the target calculation result. The target calculation result is sent to the cloud server so that the cloud server updates the stored first state data based on the target calculation result.

[0110] Optionally, the server includes a first computing unit and a second computing unit, and the processing of the to-be-processed data based on the first state data to obtain a target computing result includes:

[0111] The first computing unit processes the data to be processed using the first state data to obtain a first computing result;

[0112] Storing the first calculation result in the local database;

[0113] The second calculation unit processes the data to be processed using the first calculation result to obtain a second calculation result;

[0114] The first calculation result and the second calculation result are combined to obtain the target calculation result.

[0115] Optionally, loading the first state data in a local database of the server according to the data processing request includes:

[0116] When the first state data is loaded into the local database according to the data processing request and the loading fails, the first state data is loaded into the cloud server according to the data processing request.

[0117] Optionally, the data to be processed includes N source data, where N is an integer greater than 1. After receiving the data processing request sent by the client, the method further includes:

[0118] The N source data are merged based on a preset threshold to obtain at least one merged data, wherein the data to be processed includes the at least one merged data, and the number of source data included in the merged data is less than or equal to the preset threshold.

[0119] Optionally, the server is provided with a first logical queue and a second logical queue, and the method further includes:

[0120] receiving a capacity expansion request, the capacity expansion request being used to expand the storage capacity of the first logical queue from a first storage capacity to a second storage capacity, the first logical queue comprising at least two first physical queues;

[0121] generating a second physical queue in the server according to the capacity expansion request, wherein the storage capacity of the second physical queue is a target storage capacity, and the target storage capacity is a difference between the second storage capacity and the first storage capacity;

[0122] The second physical queue is added to the second logical queue, and the second physical queue is serially connected to the last first physical queue of the at least two first physical queues.

[0123] Optionally, the server includes a first logical queue and a second logical queue, the first logical queue includes at least two first physical queues, the second logical queue includes a third physical queue, and the third physical queue is connected in series with the last first physical queue of the at least two first physical queues, and the method further includes:

[0124] Receive a reduction request;

[0125] In response to the shrink request, the third physical queue is unbound in the second logical queue.

[0126] During the data processing process, the present application loads the first state data into the local database of the server through a data processing request, thereby processing the data to be processed through the first state data to generate a target calculation result, and thereby updating the local database and the cloud server, thereby improving the data processing efficiency.

[0127] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0128] The embodiment of the present application further provides a computer program product, which is stored in a storage medium. The computer program product is executed by at least one processor to implement the various processes of the above-mentioned data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0129] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0131] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data processing method, characterized in that: Applied to a server, the method comprises: Receiving a data processing request sent by a client, the data processing request including data to be processed, and the data processing request is used to request the server to process the data to be processed; Loading first state data in a local database of the server according to the data processing request, wherein the first state data is used to process the data to be processed; Processing the data to be processed based on the first state data to obtain a target calculation result; The target calculation result is sent to the client, and the first state data in the local database is updated based on the target calculation result. The target calculation result is sent to the cloud server so that the cloud server updates the stored first state data based on the target calculation result.

2. The method according to claim 1, characterized in that The server includes a first computing unit and a second computing unit, and the processing of the to-be-processed data based on the first state data to obtain a target computing result includes: The first computing unit processes the data to be processed using the first state data to obtain a first computing result; Storing the first calculation result in the local database; The second calculation unit processes the data to be processed using the first calculation result to obtain a second calculation result; The first calculation result and the second calculation result are combined to obtain the target calculation result.

3. The method according to claim 2, characterized in that The step of loading the first state data in the local database of the server according to the data processing request includes: When the first state data is loaded into the local database according to the data processing request and the loading fails, the first state data is loaded into the cloud server according to the data processing request.

4. The method according to claim 1, characterized in that The data to be processed includes N source data, where N is an integer greater than 1. After receiving the data processing request sent by the client, the method further includes: The N source data are merged based on a preset threshold to obtain at least one merged data, wherein the data to be processed includes the at least one merged data, and the number of source data included in the merged data is less than or equal to the preset threshold.

5. The method according to claim 1, characterized in that The server is provided with a first logical queue and a second logical queue, and the method further comprises: receiving a capacity expansion request, the capacity expansion request being used to expand the storage capacity of the first logical queue from a first storage capacity to a second storage capacity, the first logical queue comprising at least two first physical queues; generating a second physical queue in the server according to the capacity expansion request, wherein the storage capacity of the second physical queue is a target storage capacity, and the target storage capacity is a difference between the second storage capacity and the first storage capacity; The second physical queue is added to the second logical queue, and the second physical queue is serially connected to the last first physical queue of the at least two first physical queues.

6. The method according to claim 1, characterized in that The server includes a first logical queue and a second logical queue, the first logical queue includes at least two first physical queues, the second logical queue includes a third physical queue, the third physical queue is connected in series with the last first physical queue of the at least two first physical queues, and the method further includes: Receive a reduction request; In response to the shrink request, the third physical queue is unbound in the second logical queue.

7. A data processing device, characterized in that: Applied to a server, the device comprises: A receiving module, used for receiving a data processing request sent by a client, wherein the data processing request includes data to be processed, and the data processing request is used for requesting the server to process the data to be processed; A loading module, used for loading first state data in a local database of the server according to the data processing request, wherein the first state data is used for processing the data to be processed; A processing module, used for processing the data to be processed based on the first state data to obtain a target calculation result; A sending module is used to send the target calculation result to the client, update the first state data in the local database based on the target calculation result, and send the target calculation result to the cloud server so that the cloud server updates the stored first state data based on the target calculation result.

8. An electronic device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the data processing method as described in any one of claims 1 to 6.

9. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps in the data processing method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Data processing method and device

    CN110708196A

  • Data processing method and device, equipment and storage medium

    CN115145750A

  • Data storage method, device and system

    CN118244972A