Data processing method and system, computing equipment and storage medium

Abstract

The invention discloses a data processing method and system, computing equipment and a storage medium, and relates to the technical field of data processing. The method is executed in a data processing server, the data processing server comprises a scalable data pool, and the method comprises the following steps: receiving original data; judging whether the original data is business-related data, if the original data is the business-related data, sending the original data to a scalable data pool, and loading a data pool component chain; the original data is processed through a data pool assembly chain to obtain a data processing result, the data processing result is stored in a scalable data pool, and the data pool assembly chain comprises a data pool configuration assembly, a dynamic scalable assembly, a margin alarm assembly and a logic calculation assembly; and respectively sending a data processing result to a service system and a data storage device for storage. According to the technical scheme of the invention, efficient, scalable and configurable data processing and access can be realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, a data processing system, a computing device and a storage medium. Background Art

[0002] In today's big data era, the requirements for data processing are getting higher and higher, and the logic is becoming more and more complex. In most cases, the original data obtained cannot be used directly. For example, in scenarios such as synchronization and update of hot data, the frequency is high and the data volume is large. Developers need to perform refinement (judgment, calculation, merging, etc.) and processing before presenting the final results to relevant personnel. This requires the formulation of an efficient, scalable, and configurable data access strategy based on actual conditions and relevant logic.

[0003] In business systems, the commonly used and efficient data access method is the cache database, such as Redis, Memcached, etc. This method implements the basic Redis cache database, which is a high-performance key-value database storage system. It supports relatively more value types, including string, list, set, zset (sorted set-ordered set) and hash. These data types support operations such as storage, retrieval, addition, and deletion, and some operations are atomic. On this basis, in order to ensure efficiency, data is cached in memory.

[0004] In the prior art, every time the hot data changes, it will be synchronized to the business system. After the business system receives the data, it will directly update the fields that need to be extracted from the hot data to the corresponding main fields. This solution is simple and effective when the amount of data is not too large. However, this solution is not suitable for use in scenarios with large amounts of data. Every time the business system receives synchronized data, the request will be directly sent to the business database. In severe cases, the database connection pool resources will be occupied all the time, affecting the use of other business lines, causing waste of resources, and even causing the production environment to crash.

[0005] There is another solution in the prior art, which is to extract the required fields in the hot data after the business system receives the data, and store all the processed data directly in the cache according to the corresponding primary key. Each time data is received, the value of the same type of field is overwritten and updated according to the cache primary key. In addition, it is necessary to start the synchronization service to persist the cached data to the business library according to the primary key. This solution solves the problem of database connection pool resource occupation when the amount of data is large. However, this solution requires the startup of the synchronization service, the development cost is relatively high, and when the amount of data is large, the space occupied by the cache will also increase accordingly. Moreover, the implementation process of this solution is complicated.

[0006] Therefore, a data processing method is needed to solve the problems existing in the above technical solutions. Summary of the invention

[0007] To this end, the present invention provides a data processing method and a data processing system to solve or at least alleviate the above problems.

[0008] According to one aspect of the present invention, there is provided a data processing method, which is executed in a data processing server, wherein the data processing server includes a scalable data pool, and the data processing server is respectively connected to a data storage device and a business system in communication, the method comprising: receiving original data; determining whether the original data is business-related data, and if it is business-related data, then: sending the original data to the scalable data pool, and loading a data pool component chain; processing the original data through the data pool component chain to obtain a data processing result, and storing the data processing result in the scalable data pool, wherein the data pool component chain includes a data pool configuration component, a dynamic scaling component, a margin alarm component, and a logic calculation component; and sending the data processing result to the business system and the data storage device for storage.

[0009] Optionally, in the data processing method according to the present invention, the scalable data pool includes multiple access pools; the original data is processed through the data pool component chain, including: loading the configuration information of the scalable data pool through the data pool configuration component; obtaining the configuration information and monitoring the current access pool capacity through the dynamic scaling component, so that when the current access pool capacity is greater than the maximum threshold of the access pool or less than the minimum threshold of the access pool, the current access pool capacity is dynamically adjusted according to the configuration information; monitoring and recording the operating status of the current access pool through the margin alarm component, so as to alarm when the current access pool capacity is insufficient; obtaining the current available access pool through the logic calculation component, querying whether the original data exists in the current available access pool according to the primary key of the original data, if so, reading the original data from the current available access pool and performing logic calculation processing to obtain a data processing result, and writing the data processing result back to the current available access pool.

[0010] Optionally, the data processing method according to the present invention further includes: if the original data does not exist in the currently available access pool, directly performing logical calculation processing on the original data through the logical calculation component to obtain a data processing result, and storing the data processing result in the currently available access pool.

[0011] Optionally, in the data processing method according to the present invention, each of the access pools has a corresponding code, so that storage operations are performed according to the coding sequence of the access pools.

[0012] Optionally, in the data processing method according to the present invention, the data pool component chain also includes a coding and decoding component; processing the original data through the data pool component chain also includes: decoding the original data in the scalable data pool through the coding and decoding component, and encoding the data processing result so as to write the data processing result after the encoding operation back to the currently available access pool.

[0013] Optionally, in the data processing method according to the present invention, the data pool component chain also includes a recycling component; processing the original data through the data pool component chain also includes: monitoring whether the multiple access pools include a full access pool; if a full access pool is included, triggering the recycling component to perform a recycling operation on the data in the full access pool to persistently store the recycled data in the data storage device; after the recycling operation is completed, clearing the data in the full access pool.

[0014] Optionally, in the data processing method according to the present invention, before the original data is processed by the data pool component chain, it also includes: obtaining a connection pool; after the original data is processed by the data pool component chain to obtain a data processing result, it also includes: releasing the connection pool; wherein the configuration information includes: one or more of the data pool host address, password, port number, timeout period, maximum number of active connections in the connection pool, minimum number of idle connections in the connection pool, maximum number of idle connections in the connection pool, access pool KEY, current access pool KEY, maximum access pool threshold, minimum access pool threshold, access pool warning threshold, access pool capacity, and dynamic increment of access pool.

[0015] Optionally, in the data processing method according to the present invention, after receiving the original data, it also includes: asynchronously generating corresponding log data according to the original data, and sending the log data to a data storage device for storage.

[0016] Optionally, the data processing method according to the present invention further includes: if the original data is not business-related data, discarding the original data.

[0017] Optionally, in the data processing method according to the present invention, the original data includes hotspot data.

[0018] According to one aspect of the present invention, there is provided a data processing system, comprising: a data storage device; a business system; a data processing server, wherein the data processing server comprises a scalable data pool, and the data processing server is respectively communicated with the data storage device and the business system, and is suitable for executing the method as described above.

[0019] According to one aspect of the present invention, there is provided a computing device, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the data processing method as described above.

[0020] According to one aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the data processing method as described above.

[0021] According to the technical solution of the present invention, a data processing method is provided. When receiving the original data, it is determined whether the original data is business-related data. If it is business-related data, the original data is sent to a scalable data pool and the data pool component chain is loaded; the original data is processed by the data pool component chain (including a data pool configuration component, a dynamic scaling component, a surplus alarm component, and a logical calculation component) to obtain a data processing result and store the data processing result in the scalable data pool; finally, the data processing result is sent to the business system and the data storage device for storage. In this way, the original data is processed by the data pool configuration component, the dynamic scaling component, the surplus alarm component, and the logical calculation component in the data pool component chain, which can realize the scalability and configurability of the data pool, thereby realizing efficient, scalable, and configurable data processing, and synchronizing the data processing results to the business system and persistently storing them in the data storage device. Moreover, the processing method of the original data based on the data pool component chain in the present invention can reasonably store the data in different pools, which can reduce the server pressure in the case of large data volume and high concurrency. In addition, the original data is processed and accessed only when it is determined that the original data is business-related data, which can ensure the consistency of the data in the scalable data pool and the data in the business system.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other purposes, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout the present disclosure, the same reference numerals generally refer to the same parts or elements.

[0024] Figure 1 A schematic diagram of a data processing system 100 according to an embodiment of the present invention is shown;

[0025] Figure 2 A schematic diagram of a computing device 200 according to an embodiment of the present invention is shown;

[0026] Figure 3 FIG. 3 is a flow chart of a data processing method 300 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] Figure 1 FIG. 1 is a schematic diagram of a data processing system 100 according to an embodiment of the present invention. The data processing system 100 according to the present invention can process and access data in an efficient, scalable and configurable manner.

[0029] like Figure 1 As shown, the data processing system 100 includes a front end 110, a data processing server 120, a data storage device 130, and a business system 140. The data processing server 120 includes a scalable data pool 125, and the data processing server 120 is communicated with the data storage device 130 and the business system 140 respectively.

[0030] In an embodiment of the present invention, the data processing server 120 can receive the original data, determine whether the original data is business-related data, and if it is business-related data, send the original data to the scalable data pool 125, and load the data pool component chain to process the original data through the data pool component chain. Specifically, the original data is processed through the data pool component chain to obtain a data processing result, and the data processing result can be stored in the scalable data pool 125.

[0031] In an embodiment of the present invention, the data pool component chain may include a data pool configuration component, a dynamic scaling component, a margin alarm component, and a logic calculation component.

[0032] In addition, the data processing server 120 may send the data processing results to the business system 140 and the data storage device 130 for storage, respectively, so as to synchronize the data processing results to the business system 140 and store the data processing results persistently in the data storage device 130 .

[0033] In one embodiment, the data processing server 120 may receive raw data from the front end 110, for example, may receive an update request or synchronization request for hotspot data from the front end 110. Here, the raw data may be, for example, various types of hotspot data.

[0034] The front end 110 may be a terminal device used by a user, which may be a personal computer such as a desktop computer or a laptop computer, or a mobile terminal such as a mobile phone, a tablet computer, a multimedia device, or a smart wearable device, but is not limited thereto. The front end 110 may also be an application program resident in the terminal device.

[0035] The data processing server 120 is used to provide data processing services, and can be implemented as any server capable of providing data processing services in the prior art, and the present invention does not limit the specific type of the server. For example, the data processing server 120 can be implemented as a desktop computer, a laptop computer, a processor chip, a mobile phone, a tablet computer, etc., but is not limited thereto. The server can also be a service program resident in a computing device.

[0036] In an embodiment of the present invention, the data processing server 120 may be configured to execute the data processing method 300 to process and access the original data, synchronize the data processing results to the business system 140, and store them persistently in the data storage device 130. The data processing method 300 of the present invention will be described in detail below.

[0037] In one embodiment, the data processing server 120 of the present invention may be implemented as a computing device 200 as described below, so that the data processing method 300 of the present invention may be executed in the computing device 200 .

[0038] Figure 2 FIG. 2 shows a block diagram of a computing device 200 according to an embodiment of the present invention. Figure 2 As shown, in a basic configuration 202, computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between processor 204 and system memory 206.

[0039] Depending on the desired configuration, the processor 204 can be any type of processor, including but not limited to: a microprocessor (μP), a microcontroller (μC), a digital information processor (DSP), or any combination thereof. The processor 204 can include one or more levels of cache such as a primary cache 210 and a secondary cache 212, a processor core 214, and registers 216. An example processor core 214 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 218 can be used with the processor 204, or in some implementations, the memory controller 218 can be an internal part of the processor 204.

[0040] Depending on the desired configuration, the system memory 206 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The system memory 106 may include an operating system 220, one or more applications 222, and program data 224. The application 222 is actually a plurality of program instructions, which are used to instruct the processor 204 to perform corresponding operations. In some embodiments, the application 222 can be arranged to cause the processor 204 to operate using the program data 224 on the operating system.

[0041] The computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 250. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 258. Example communication devices 246 may include a network controller 260, which may be arranged to facilitate communication with one or more other computing devices 262 via a network communication link via one or more communication ports 264.

[0042] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A "modulated data signal" can be a signal in which one or more of its data sets or its changes can be carried out in a manner that encodes information in the signal. As non-limiting examples, communication media can include wired media such as a wired network or a dedicated line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR) or other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0043] In the computing device 200 according to the present invention, the application 222 of the computing device 200 includes multiple program instructions for executing the data processing method 300. These program instructions can instruct the processor 204 to execute the data processing method 300 of the present invention, so that the computing device 200 can realize efficient, scalable and configurable processing and access to data by executing the data processing method 300 of the present invention.

[0044] Figure 3 A flow chart of a data processing method 300 according to an embodiment of the present invention is shown. The method 300 is suitable for being executed in a data processing server 120 (e.g., the aforementioned computing device 200). The data processing server 120 includes a scalable data pool 125, and the data processing server 120 is respectively connected to a data storage device 130 and a business system 140 for communication.

[0045] like Figure 3 As shown, method 300 begins at step 310 .

[0046] In step 310 , the data processing server 120 receives the original data. Here, the data processing server 120 may be connected to the front end 110 for communication, and may receive the original data sent by the front end 110 .

[0047] Here, in some embodiments, the original data may be, for example, various types of hotspot data. In step 310, what is received may be an update request or synchronization request of the front end 110 for the hotspot data.

[0048] In some embodiments, after receiving the original data, a single thread may be used to asynchronously generate corresponding log data according to the original data, and the generated log data may be sent to the data storage device 130 for storage. In addition, the original data may be verified according to pre-configured verification rules, so that after the verification is passed, the following step 320 is performed.

[0049] Subsequently, in step 320, it is determined whether the original data is business-related data. If it is business-related data, the following steps 330 to 350 are executed to ensure that only business-related data is processed and accessed, ensuring the consistency of the data in the scalable data pool 125 with the data in the business system 140.

[0050] It should be noted that business-related data refers to data related to the business processed by the business system 140 .

[0051] In step 330, the raw data is sent to the scalable data pool 125, and the data pool component chain is loaded.

[0052] In some embodiments, before the raw data is sent to the scalable data pool 125, the raw data may be parsed into data in a predetermined format to facilitate subsequent processing of the data in the predetermined format. For example, the raw data may be parsed into data in a json format using a json serialization method, but the present invention is not limited to this parsing method, and the specific parsing logic may be set according to the actual application scenario.

[0053] Next, in step 340 , the original data is processed (eg, including operations such as judgment, calculation, and merging) through the data pool component chain to obtain a data processing result, and the data processing result is stored in the scalable data pool 125 .

[0054] In an embodiment of the present invention, the data pool component chain may include a data pool configuration component, a dynamic scaling component, a margin alarm component, and a logic calculation component. Thus, in step 340, the data pool configuration component, the dynamic scaling component, the margin alarm component, and the logic calculation component may be used to complete the processing of the original data, obtain the data processing result, and store the data processing result in the scalable data pool 125.

[0055] It should be noted that the scalable data pool 125 includes multiple access pools. In some embodiments, each access pool has a corresponding code, so that in the scalable data pool 125, data can be stored according to the coding order of the access pool.

[0056] In a specific embodiment, processing the original data through the data pool component chain may specifically include the following steps.

[0057] Through the data pool configuration component, the configuration information of the scalable data pool 125 is loaded. In some embodiments, the configuration information may include: one or more of the data pool host address, password, port number, timeout period, maximum number of active connections in the connection pool, minimum number of idle connections in the connection pool, maximum number of idle connections in the connection pool, access pool KEY, current access pool KEY, maximum access pool threshold, minimum access pool threshold, access pool warning threshold, access pool capacity, and dynamic increment of access pool.

[0058] The configuration information is obtained through the dynamic scaling component to monitor the current access pool capacity. When it is detected that the current access pool capacity is greater than the maximum access pool threshold or less than the minimum access pool threshold, the current access pool capacity can be dynamically adjusted according to the configuration information (dynamic increment of the access pool). Here, in one embodiment, a single thread can be used to monitor the current access pool capacity regularly.

[0059] The current access pool operation status is monitored and recorded by the surplus alarm component, so as to alarm when the current access pool capacity is insufficient. In this way, the current access pool capacity can be dynamically adjusted according to the alarm information. Here, in one embodiment, a single thread can be used to regularly monitor the current access pool operation status and record it.

[0060] Through the logic calculation component, the current available access pool is obtained, and the primary key of the original data is used to query whether the original data exists in the current available access pool. If the original data exists in the current available access pool, the original data is read from the current available access pool, and the read original data is logically calculated to obtain a data processing result, and then the data processing result can be written back to the current available access pool. Here, the specific calculation logic for the logical calculation of the original data can be customized by those skilled in the art according to actual needs.

[0061] In addition, if the original data does not exist in the current available access pool, the original data received by the scalable data pool 125 (in step 330) can be directly processed by logical calculation through the logical calculation component to obtain a data processing result, and the data processing result is stored in the current available access pool.

[0062] In this way, the original data is processed through the data pool configuration component, dynamic scaling component, surplus alarm component, and logic calculation component in the data pool component chain, so that the data pool can be scalable and configurable.

[0063] It should be noted that the present invention can determine the number and capacity of access pools based on the amount of business data. According to the method of processing original data based on the data pool component chain in the present invention, the data can be reasonably stored in pools, which can reduce the server pressure in cases of large data volume and high concurrency.

[0064] In some embodiments, the data pool component chain may further include a coding and decoding component. Thus, in the process of processing the original data through the data pool component chain, the coding and decoding component may also be used to perform a decoding operation on the original data in the scalable data pool 125. Furthermore, the coding and decoding component may be used to perform an encoding operation on the processed data processing result, so as to write the data processing result after the encoding operation back to the currently available access pool.

[0065] Finally, in step 350 , the data processing results are sent to the business system 140 and the data storage device 130 for storage, so as to synchronize the data processing results to the business system 140 and store the data processing results persistently in the data storage device 130 .

[0066] In addition, in step 320, it is determined whether the original data is business-related data. If it is business-related data, steps 330 to 350 are executed. If it is determined that the original data is not business-related data, the original data is discarded and no subsequent data processing is performed on the original data. Only when the original data is determined to be business-related data, the original data is processed and accessed, which can ensure the consistency of the data in the scalable data pool 125 and the data in the business system 140.

[0067] In some embodiments of the present invention, the data pool component chain further includes a recycling component, through which the data fetching operation can be implemented. Specifically, in step 340, the raw data is processed through the data pool component chain, and the following steps may also be included.

[0068] Monitor whether a plurality of access pools includes a full access pool. Here, in one embodiment, a single thread may be used to periodically monitor whether a plurality of access pools includes a full access pool.

[0069] If a full access pool is included, the recycling component is triggered to perform a recycling operation on the data in the full access pool, so as to persistently store the recycled data in the data storage device 130. Furthermore, after the recycling operation is completed, the data in the full access pool is cleared.

[0070] In some embodiments, before the original data is processed by the data pool component chain, a connection pool may be obtained. After the original data is processed by the data pool component chain to obtain the data processing result, the connection pool may be released.

[0071] In the above embodiments, different threads in the thread pool may be used to implement different operations, supporting multi-threaded processing, thereby achieving parallel and efficient data processing and access operations.

[0072] It should be noted that, in the embodiment of the present invention, the scalable data pool 125 is suitable for using the Redis access method. That is, through the data pool component chain, the Redis access method can be used to implement data access operations (storage operations for storing data in the access pool and retrieval operations for obtaining data from the access pool), so that the connection pool resource occupation problem caused by large data volume and high concurrency can be solved. In addition, by using Redis distributed locks to implement access operations, the accuracy of data access operations can be guaranteed.

[0073] According to the data processing method 300 of the present invention, when the original data is received, it is determined whether the original data is business-related data. If it is business-related data, the original data is sent to the scalable data pool and the data pool component chain is loaded; the original data is processed by the data pool component chain (including the data pool configuration component, the dynamic scaling component, the surplus alarm component, and the logical calculation component) to obtain the data processing result and store the data processing result in the scalable data pool; finally, the data processing result is sent to the business system and the data storage device for storage. In this way, the original data is processed by the data pool configuration component, the dynamic scaling component, the surplus alarm component, and the logical calculation component in the data pool component chain, which can realize the scalability and configurability of the data pool, thereby realizing efficient, scalable, and configurable processing of data, and synchronizing the data processing results to the business system and persistently storing them in the data storage device. Moreover, the processing method of the original data based on the data pool component chain in the present invention can reasonably store the data in different pools, which can reduce the server pressure in the case of large data volume and high concurrency. In addition, the original data is processed and accessed only when it is determined that the original data is business-related data, which can ensure the consistency of the data in the scalable data pool and the data in the business system.

[0074] A8. A method as described in any one of A1-A7, wherein after receiving the original data, it also includes: asynchronously generating corresponding log data according to the original data, and sending the log data to a data storage device for storage.

[0075] A9. A method as described in any one of A1-A8, further comprising: if the original data is not business-related data, discarding the original data.

[0076] A10. A method as described in any one of A1-A8, wherein the original data includes hotspot data.

[0077] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or parts of the method and apparatus of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0078] In the case where the program code is executed on a programmable computer, the mobile terminal generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the data processing method of the present invention according to the instructions in the program code stored in the memory.

[0079] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0080] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to implement the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0081] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0082] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof.

[0083] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0084] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of submodules or subunits or subcomponents.

[0085] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0086] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0087] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0088] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is primarily selected for readability and instructional purposes, rather than for explaining or limiting the subject matter of the present invention.

Claims

1. A data processing method is executed in a data processing server, wherein the data processing server includes a scalable data pool, and the data processing server is respectively connected to a data storage device and a business system in communication, the method comprising: Receive raw data; Determine whether the original data is business-related data. If it is business-related data, then: Sending the raw data to the scalable data pool and loading the data pool component chain; Processing the raw data through the data pool component chain to obtain a data processing result, and storing the data processing result in the scalable data pool, wherein the data pool component chain includes a data pool configuration component, a dynamic scaling component, a margin alarm component, and a logic calculation component; The data processing results are sent to the business system and the data storage device for storage.

2. The method of claim 1, wherein: The scalable data pool includes a plurality of access pools; the raw data is processed through the data pool component chain, including: Loading configuration information of a scalable data pool through the data pool configuration component; The configuration information is obtained through the dynamic scaling component, and the current access pool capacity is monitored, so that when the current access pool capacity is greater than a maximum access pool threshold or less than a minimum access pool threshold, the current access pool capacity is dynamically adjusted according to the configuration information; The remaining capacity alarm component is used to monitor and record the current access pool operation status, so as to generate an alarm when the current access pool capacity is insufficient; Through the logical computing component, the currently available access pool is obtained, and whether the original data exists in the currently available access pool is queried based on the primary key of the original data. If so, the original data is read from the currently available access pool and logical computing is performed to obtain a data processing result, and the data processing result is written back to the currently available access pool.

3. The method of claim 2, wherein: Also includes: If the original data does not exist in the currently available access pool, the logic calculation component directly performs logic calculation processing on the original data to obtain a data processing result, and the data processing result is stored in the currently available access pool.

4. The method according to claim 2 or 3, wherein: Each of the access pools has a corresponding code, so that storage operations are performed according to the code sequence of the access pools.

5. The method according to any one of claims 2 to 4, wherein: The data pool component chain also includes a coding and decoding component; processing the original data through the data pool component chain also includes: The encoding and decoding component is used to perform a decoding operation on the original data in the scalable data pool, and an encoding operation is performed on the data processing result, so as to write the data processing result after the encoding operation back to the currently available access pool.

6. The method according to any one of claims 2 to 5, wherein: The data pool component chain further includes a recycling component; and processing the original data through the data pool component chain further includes: Monitoring whether the multiple access pools include a full access pool; If a full access pool is included, triggering the recycling component to perform a recycling operation on the data in the full access pool, so as to persistently store the recycled data in the data storage device; After the recycling operation is completed, the data in the full access pool is cleared.

7. The method according to any one of claims 2 to 6, wherein: Before the raw data is processed by the data pool component chain, the method further includes: obtaining a connection pool; After the original data is processed by the data pool component chain to obtain a data processing result, the method further includes: releasing the connection pool; Among them, the configuration information includes: one or more of: data pool host address, password, port number, timeout period, maximum number of active connections in the connection pool, minimum number of idle connections in the connection pool, maximum number of idle connections in the connection pool, access pool KEY, current access pool KEY, maximum access pool threshold, minimum access pool threshold, access pool warning threshold, access pool capacity, and dynamic increment of access pool.

8. A data processing system comprising: Data storage devices; Business systems; A data processing server, wherein the data processing server comprises a scalable data pool, and the data processing server is respectively communicated with a data storage device and a business system, and is suitable for executing the method as described in any one of claims 1-7.

9. A computing device comprising: at least one processor; and A memory storing program instructions, wherein the program instructions are configured to be processed by the at least one processor, and the program instructions include instructions for processing the method according to any one of claims 1 to 7.

10. A readable storage medium storing program instructions, which, when read and processed by a computing device, enables the computing device to process the method according to any one of claims 1 to 7.

Images