Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the following steps: receiving a plurality of pieces of data to be cached, and obtaining a current fragment number from a distributed configuration center; wherein the to-be-cached data indicates a data number; according to the current fragment number and the data number, performing fragment processing on the multiple pieces of to-be-cached data, and caching the multiple pieces of to-be-cached data subjected to fragment processing into cache middleware; after the cache middleware is successfully cached, calling an interface of the distributed configuration center to send a cache notification to a local cache, so as to synchronously cache the data cached in the cache middleware into the local cache; wherein the cache data in the local cache is a resident memory object. According to the embodiment, the data processing efficiency is improved, the cache performance is guaranteed, the application scene of data processing is expanded, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method and device. Background Art

[0002] With the development of Internet technology, Internet business has also grown rapidly. How to cache and process the large amount of business data generated to cope with subsequent data query operations is an urgent problem that needs to be solved. In the related art, data caching operations are mainly performed through a secondary cache mechanism composed of a remote cache and a local cache. However, when the remote cache and the local cache are used at the same time, there is a high degree of complexity in cache maintenance, and when the local cache fails, the business will be unavailable. On the other hand, in the related art, information notification between the remote cache and the local cache is generally performed by introducing a message middleware, which requires adjustments to the original architecture of the system, increasing the complexity of the system architecture. At the same time, the existing cache mechanism often adopts a key-value storage structure. When faced with millions of data caching scenarios, a large number of keys reduces cache performance, affects data caching and query efficiency, and leads to low data processing efficiency and poor user experience. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a data processing method and device, which can first perform shard storage on the cached data through the cache middleware, and then use the local cache to set the cached data as a resident memory object, thereby avoiding the situation where the business is unavailable when the local cache fails. At the same time, the distributed configuration center executes message notifications between the cache middleware and the local cache, which can be applicable to high-concurrency data caching scenarios, improves data processing efficiency, ensures cache performance, expands the application scenarios of data processing, and enhances user experience.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0005] Receive multiple data to be cached and obtain the current number of shards from the distributed configuration center; the data to be cached indicates the data number;

[0006] Segment multiple cached data according to the current shard number and data number, and cache the multiple cached data after sharding into the cache middleware;

[0007] After the cache middleware successfully caches, it calls the interface of the distributed configuration center to send a cache notification to the local cache to synchronize the data cached in the cache middleware to the local cache; among them, the cached data in the local cache is a resident memory object.

[0008] Furthermore, the step of sharding the plurality of data to be cached according to the current number of shards and the data serial number includes:

[0009] Perform a modulo operation on the data number using the value corresponding to the current number of shards as the modulo;

[0010] The shard number corresponding to each to-be-cached data is determined according to the modulo operation result, so as to implement shard processing of multiple to-be-cached data.

[0011] Furthermore, the data to be cached further indicates a data type; the step of caching the plurality of data to be cached after sharding into the cache middleware includes:

[0012] The multiple cached data after sharding are cached in the cache middleware in the form of key-value pairs; wherein the key in the key-value pair includes the data type and the shard number, and the key value in the key-value pair includes the cached data stored in the corresponding shard.

[0013] Furthermore, after the step of obtaining the current number of shards from the distributed configuration center, the method further includes:

[0014] Determine whether to expand the current number of shards based on the remaining cache space for the current shards and the amount of data to be cached.

[0015] If so, determine the actual number of shards after expansion processing based on the remaining cache space and the data volume corresponding to the multiple data to be cached, and shard the multiple data to be cached according to the actual number of shards and the data number, and cache the multiple data to be cached after sharding in the cache middleware.

[0016] Furthermore, the method further comprises:

[0017] Using the value corresponding to the actual number of shards after the expansion process as the modulus, perform a modulo operation on the data number of the cached data in the cache middleware, and determine the actual shard number corresponding to the cached data based on the modulo operation result to update the cached data;

[0018] And send the updated results to the local cache.

[0019] Furthermore, the step of receiving multiple data to be cached and obtaining the current number of shards from the distributed configuration center also includes:

[0020] Batch receive multiple data to be cached, and perform verification processing on the multiple data to be cached;

[0021] If the verification passes, obtain the current number of shards from the distributed configuration center.

[0022] Furthermore, the method further comprises:

[0023] Receive a target data query request, and determine a shard number corresponding to the target data according to the data number of the target data indicated by the target data query request;

[0024] The local cache is queried based on the data number and shard number, and the query result is fed back to the requester of the target data query request.

[0025] According to another aspect of an embodiment of the present invention, there is provided a data processing apparatus, including:

[0026] A receiving module is used to receive multiple data to be cached and obtain the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number;

[0027] The sharding module is used to shard the multiple cached data according to the current shard number and data number, and cache the multiple cached data after sharding into the cache middleware;

[0028] The data processing module is used to call the interface of the distributed configuration center to send a cache notification to the local cache after the cache middleware successfully caches the data, so as to synchronize the cached data in the cache middleware to the local cache; wherein the cached data in the local cache is a resident memory object.

[0029] According to another aspect of an embodiment of the present invention, there is provided an electronic device for data processing, comprising:

[0030] one or more processors;

[0031] a storage device for storing one or more programs,

[0032] When one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned data processing methods.

[0033] According to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above-mentioned data processing methods is implemented.

[0034] One embodiment of the above invention has the following advantages or beneficial effects: because it adopts the method of receiving multiple data to be cached, obtaining the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number; sharding the multiple data to be cached according to the current number of shards and the data number, and caching the multiple data to be cached after sharding into the cache middleware; after the cache middleware caches successfully, calling the interface of the distributed configuration center to send a cache notification to the local cache, so as to synchronously cache the data cached in the cache middleware into the local cache; wherein the cached data in the local cache is a technical means of resident memory objects, so it overcomes the cache maintenance problem existing in the related art. It solves the technical problems of high complexity, business unavailability when local cache fails, high complexity of system architecture, low data processing efficiency, poor user experience, and difficulty in applying to high-concurrency data caching scenarios. It can first use the cache middleware to store cached data in shards, and then use the local cache to set the cached data as a resident memory object, avoiding the situation where business is unavailable when the local cache fails. At the same time, the message notification between the cache middleware and the local cache is carried out through the distributed configuration center, which can be applied to high-concurrency data caching scenarios, improves data processing efficiency, ensures cache performance, expands the application scenarios of data processing, and improves the technical effect of user experience.

[0035] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0037] Figure 1 is a schematic diagram of the main flow of a data processing method provided according to one embodiment of the present invention;

[0038] Figure 2a is a schematic diagram of the main flow of a data processing method provided according to another embodiment of the present invention;

[0039] Figure 2b yes Figure 2a Schematic diagram of sharding cached data by modulo operation in the method shown;

[0040] Figure 3 is a schematic diagram of main modules of a data processing device provided according to an embodiment of the present invention;

[0041] Figure 4 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0042] Figure 5It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved in the data processing process are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0045] To collect statistics on the use of our products and services, we aggregate, analyze, and use technically processed user data and share the processed statistical information with third parties. We use secure encryption and other methods to ensure that the recipient of the information cannot re-identify specific individuals.

[0046] Figure 1 FIG. 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention; FIG. Figure 1 As shown, the data processing method provided by the embodiment of the present invention mainly includes:

[0047] Step S101: Receive multiple data to be cached and obtain the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number.

[0048] Specifically, according to an embodiment of the present invention, the data to be cached may include data of multiple business types. Accordingly, if multiple business types are involved, the data to be cached may also indicate the corresponding business types. Depending on the business types of the data to be cached, the data formats of the multiple data to be cached may also be inconsistent. For example, if the data to be cached is a list of SKUs (StockKeeping Units, the unit of measurement for inventory in and out), multiple table files corresponding to the data to be cached may be obtained.

[0049] The distributed configuration center is used to configure the number of shards corresponding to the cache, execute message notifications between the cache middleware and the local cache, etc.

[0050] Furthermore, according to an embodiment of the present invention, the step of receiving a plurality of data to be cached and obtaining the current number of shards from the distributed configuration center further includes:

[0051] Batch receive multiple data to be cached, and perform verification processing on the multiple data to be cached;

[0052] If the verification passes, obtain the current number of shards from the distributed configuration center.

[0053] By verifying multiple cached data to filter out erroneous, duplicate, and existing cached data, it helps save cache space and improve subsequent data processing efficiency.

[0054] Step S102 : Slice the plurality of data to be cached according to the current number of shards and the data serial number, and cache the plurality of data to be cached after the slicing process into the cache middleware.

[0055] Specifically, according to an embodiment of the present invention, the step of sharding the plurality of data to be cached according to the current number of shards and the data serial number includes:

[0056] Perform a modulo operation on the data number using the value corresponding to the current number of shards as the modulo;

[0057] The shard number corresponding to each to-be-cached data is determined according to the modulo operation result, so as to implement shard processing of multiple to-be-cached data.

[0058] Through the above settings, the data numbers of multiple data to be cached are modulo the value of the current number of shards, so as to realize sharding processing of multiple data to be cached, effectively reducing the amount of data that needs to be processed subsequently and improving cache efficiency.

[0059] Furthermore, according to an embodiment of the present invention, the data to be cached further indicates a data type; and the step of caching the plurality of data to be cached after sharding into the cache middleware includes:

[0060] The multiple cached data after sharding are cached in the cache middleware in the form of key-value pairs; wherein the key in the key-value pair includes the data type and the shard number, and the key value in the key-value pair includes the cached data stored in the corresponding shard.

[0061] Through the above settings, the data type and shard number are used as the key in the key-value pair, and the cached data in the shard is used as the corresponding value. This greatly reduces the number of keys required to set during the caching process, ensures cache performance, improves the efficiency of data caching and subsequent data queries, and is suitable for high-concurrency data caching scenarios.

[0062] Optionally, according to an embodiment of the present invention, the data type indicated by the data to be cached in the above-mentioned key is equivalent to a key prefix. If the business type is indicated in the data to be cached, the business type can also be used as a key prefix, that is, the business type and the shard number are used as the key in the key-value pair. According to a specific implementation of an embodiment of the present invention, the key prefix in the key-value pair may include the current system time or the current cache time in addition to the business type or data type. Through the above-mentioned setting, while reducing the number of keys, the key prefix of the same batch of data to be cached is set, which is convenient for further caching of data in the shard in the future, avoiding key conflicts, ensuring cache performance, and improving user experience.

[0063] Preferably, according to an embodiment of the present invention, after the above step of obtaining the current number of shards from the distributed configuration center, the method further includes:

[0064] Determine whether to expand the current number of shards based on the remaining cache space for the current shards and the amount of data to be cached.

[0065] If so, determine the actual number of shards after expansion processing based on the remaining cache space and the data volume corresponding to the multiple data to be cached, and shard the multiple data to be cached according to the actual number of shards and the data number, and cache the multiple data to be cached after sharding in the cache middleware.

[0066] Through the above settings, combined with the actual amount of data to be cached, the number of current shards and the remaining cache space, the number of shards in the cache middleware can be dynamically adjusted, further improving data processing efficiency and expanding the application scenarios of data processing.

[0067] Illustratively, according to an embodiment of the present invention, the method further includes:

[0068] Using the value corresponding to the actual number of shards after the expansion process as the modulus, perform a modulo operation on the data number of the cached data in the cache middleware, and determine the actual shard number corresponding to the cached data based on the modulo operation result to update the cached data;

[0069] And send the updated results to the local cache.

[0070] To ensure the consistency of the local cache and the cached data in the cache middleware, it is necessary to synchronize and update the local cache in a timely manner after the cached data and the key corresponding to the cached data in the cache middleware are successfully changed.

[0071] Step S103, after the cache middleware caches successfully, calls the interface of the distributed configuration center to send a cache notification to the local cache to synchronize the data cached in the cache middleware to the local cache; wherein the cached data in the local cache is a resident memory object.

[0072] Through the above settings, the distributed configuration center replaces the existing message middleware, realizing message notifications between the cache middleware and the local cache. This avoids the need for the message middleware to adjust the original system architecture, which increases the complexity of the system architecture. It also provides a second-level response to message notifications, allowing the local cache to obtain the latest cache for synchronization after the cache middleware successfully caches. At the same time, because the cached data in the local cache is a resident memory object, it avoids the situation where the business is unavailable when the local cache fails, ensuring cache performance and expanding the application scenarios of data processing.

[0073] According to one specific implementation of an embodiment of the present invention, the local cache is a cache inside the JVM (Java Virtual Machine), and the cached data in the local cache is set as a strong reference object, so that even if the current memory is insufficient, the cached data will not be recycled, ensuring that the cached data in the local cache remains resident. According to another specific implementation of an embodiment of the present invention, it is also possible to divide the local cache into multiple partitions and place the cached data in a specific partition to ensure that the cached data remains resident, so as to avoid the situation where the service is unavailable when the local cache fails.

[0074] Furthermore, according to an embodiment of the present invention, the method further includes:

[0075] Receive a target data query request, and determine a shard number corresponding to the target data according to the data number of the target data indicated by the target data query request;

[0076] The local cache is queried based on the data number and shard number, and the query result is fed back to the requester of the target data query request.

[0077] Through the above arrangement, in response to a query request, the query result can be determined quickly and accurately based on the data number.

[0078] According to the technical solution of the embodiment of the present invention, because it adopts the method of receiving multiple data to be cached, obtaining the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number; sharding the multiple data to be cached according to the current number of shards and the data number, and caching the multiple data to be cached after sharding into the cache middleware; after the cache middleware caches successfully, calling the interface of the distributed configuration center to send a cache notification to the local cache, so as to synchronously cache the data cached in the cache middleware into the local cache; wherein the cached data in the local cache is a technical means of resident memory objects, so it overcomes the high complexity of cache maintenance existing in the related art. , when the local cache fails, it will lead to business unavailability, high complexity of system architecture, low data processing efficiency, poor user experience, and it is difficult to apply to high-concurrency data caching scenarios. Technical problems, thereby achieving the ability to first use the cache middleware to store the cached data in shards, and then use the local cache to set the cached data as a resident memory object, avoiding the situation where the business is unavailable when the local cache fails. At the same time, the message notification between the cache middleware and the local cache is carried out through the distributed configuration center, which can be applied to high-concurrency data caching scenarios, improves data processing efficiency, ensures cache performance, expands the application scenarios of data processing, and improves the technical effect of user experience.

[0079] Figure 2a is a schematic diagram of the main flow of a data processing method provided according to another embodiment of the present invention; Figure 2a As shown, taking the data to be cached as the sku whitelist as an example (it should be noted that the data to be cached as sku is only an example and is not intended to limit the present application), the data processing method provided in the embodiment of the present invention mainly includes:

[0080] Step S201, batch receive multiple sku whitelists, and perform verification on multiple data to be cached; if the verification passes, obtain the current number of shards from the distributed configuration center; wherein the sku whitelist indicates the sku number.

[0081] Depending on the business type of the data to be cached, the data formats of the multiple data to be cached may also be different. For example, if the data to be cached is a list of SKUs (Stock Keeping Units), multiple table files corresponding to the data to be cached may be obtained. Accordingly, depending on the actual situation, the multiple data to be cached may also be stored in the form of documents, compressed packages, etc.

[0082] If the multiple SKU whitelists obtained correspond to table files, the multiple SKU whitelist data can be read by parsing the table files, and then the multiple SKU whitelists can be screened out for errors, duplications, and existing SKU whitelists to determine the SKU whitelist that ultimately needs to be cached.

[0083] By verifying multiple cached data to filter out erroneous, duplicate, and existing cached data, it helps save cache space and improve subsequent data processing efficiency.

[0084] According to an embodiment of the present invention, the distributed configuration center can adopt existing technologies, such as DUCC (Distributed Unified Configuration Center). Before receiving multiple data to be cached, the distributed configuration center can configure the initial number of shards as the current number of shards based on historical cache information.

[0085] Step S202: perform a modulo operation on the SKU number with the value corresponding to the current number of fragments as the modulus; determine the fragment number corresponding to each SKU whitelist according to the modulo operation result to implement fragment processing of multiple SKU whitelists.

[0086] Through the above settings, the SKU numbers of multiple SKU whitelists are modulo the value of the current number of shards to achieve sharding processing of multiple SKU whitelists, effectively reducing the amount of data that needs to be processed subsequently and improving cache efficiency.

[0087] like Figure 2b The diagram shows how to shard multiple SKU whitelists using a modulo operation, taking the current number of shards as 1000 as an example. Since SKU is a Long value, a SKU converted to a string with a length of 20 bytes roughly equals 20 bytes. A 1KB value can store 500 SKUs. Without affecting the performance of the cache middleware, a single collection (i.e., key-value) can store 1000 SKUs. This means that a whitelist of 1 million SKUs only requires 1000 keys, reducing the number of keys by a factor of 1000.

[0088] Step S203, cache the multiple cached data after sharding in the cache middleware in the form of key-value pairs; wherein the key in the key-value pair includes the data type and the shard number, and the key value in the key-value pair includes the cached data stored in the corresponding shard.

[0089] Through the above settings, the data type and SKU number are used as the key in the key-value pair, and the SKU whitelist cached in the shard is used as the corresponding value. This greatly reduces the number of keys required to be set during the caching process, ensures cache performance, improves data caching and subsequent data query efficiency, and is suitable for high-concurrency data caching scenarios.

[0090] Optionally, according to an embodiment of the present invention, the data type indicated by the data to be cached in the above-mentioned key is equivalent to the key prefix, and the shard number is equivalent to the key index. If the business type is indicated in the data to be cached, the business type can also be used as the key prefix, that is, the business type and the shard number are used as the key in the key-value pair. According to a specific implementation of an embodiment of the present invention, the key prefix in the key-value pair may include the current system time or the current cache time in addition to the business type or data type. Through the above-mentioned setting, while reducing the number of keys, the key prefix of the same batch of data to be cached is set, which is convenient for further caching of data in the shard in the future, avoiding key conflicts, ensuring cache performance, and improving user experience.

[0091] According to a specific implementation of an embodiment of the present invention, the cache key uses a prefix + shard index. The prefix can be fixed as "white_+current system time." For example, if a cache write is performed at "2022-11-2 208:11:22," the prefix is ​​"white_20221122081122_," the number of shards is 1000, and the shard index is 0-999 or 1-1000. The minimum key for a cache shard can be expressed as "white_20221122081122_0," and the corresponding maximum key for a cache shard is expressed as "white_20221122081122_999." After determining the shard number and key-value pair corresponding to each SKU, batch caching middleware can be used.

[0092] Preferably, according to an embodiment of the present invention, after the above step of obtaining the current number of shards from the distributed configuration center, the method further includes:

[0093] Determine whether to expand the current number of shards based on the remaining cache space for the current shards and the amount of data to be cached.

[0094] If so, determine the actual number of shards after expansion processing based on the remaining cache space and the data volume corresponding to the multiple data to be cached, and shard the multiple data to be cached according to the actual number of shards and the data number, and cache the multiple data to be cached after sharding in the cache middleware.

[0095] Through the above settings, combined with the actual amount of data to be cached, the number of current shards and the remaining cache space, the number of shards in the cache middleware can be dynamically adjusted, further improving data processing efficiency and expanding the application scenarios of data processing.

[0096] Illustratively, according to an embodiment of the present invention, the method further includes:

[0097] Using the value corresponding to the actual number of shards after the expansion process as the modulus, perform a modulo operation on the data number of the cached data in the cache middleware, and determine the actual shard number corresponding to the cached data based on the modulo operation result to update the cached data;

[0098] And send the updated results to the local cache.

[0099] To ensure the consistency of the local cache and the cached data in the cache middleware, it is necessary to synchronize and update the local cache in a timely manner after the cached data and the key corresponding to the cached data in the cache middleware are successfully changed.

[0100] Step S204, after the cache middleware successfully caches, calls the interface of the distributed configuration center to send a cache notification to the local cache to synchronize the data cached in the cache middleware to the local cache; wherein, the local cache is the cache inside the JVM, and the cached data is a strong reference object of JAVA.

[0101] Through the above settings, the cached data in the local cache is set as a strong reference object, so that even if the current memory is insufficient, the cached data will not be recycled by the JVM, ensuring that the cached data in the local cache remains resident. At the same time, the distributed configuration center replaces the message middleware in the existing technology to realize message notification between the cache middleware and the local cache, that is, avoiding the situation where the message middleware needs to adjust the original system architecture and increase the complexity of the system architecture. It can also respond to message notifications in seconds, so that after the cache middleware successfully caches, the local cache can promptly obtain the latest cache for synchronization.

[0102] Step S205, receive a target data query request, determine the shard number corresponding to the target data according to the data number of the target data indicated by the target data query request; query the local cache according to the data number and shard number, and feed back the query result to the requester of the target data query request.

[0103] Through the above arrangement, in response to a query request, the query result can be determined quickly and accurately based on the data number.

[0104] According to the technical solution of the embodiment of the present invention, because it adopts the method of receiving multiple data to be cached, obtaining the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number; sharding the multiple data to be cached according to the current number of shards and the data number, and caching the multiple data to be cached after sharding into the cache middleware; after the cache middleware caches successfully, calling the interface of the distributed configuration center to send a cache notification to the local cache, so as to synchronously cache the data cached in the cache middleware into the local cache; wherein the cached data in the local cache is a technical means of resident memory objects, so it overcomes the high complexity of cache maintenance existing in the related art. , when the local cache fails, it will lead to business unavailability, high complexity of system architecture, low data processing efficiency, poor user experience, and it is difficult to apply to high-concurrency data caching scenarios. Technical problems, thereby achieving the ability to first use the cache middleware to store the cached data in shards, and then use the local cache to set the cached data as a resident memory object, avoiding the situation where the business is unavailable when the local cache fails. At the same time, the message notification between the cache middleware and the local cache is carried out through the distributed configuration center, which can be applied to high-concurrency data caching scenarios, improves data processing efficiency, ensures cache performance, expands the application scenarios of data processing, and improves the technical effect of user experience.

[0105] Figure 3 Schematic diagram of the main modules of the data processing device provided according to an embodiment of the present invention; Figure 3 As shown, the data processing device 300 provided in this embodiment of the present invention mainly includes:

[0106] The receiving module 301 is used to receive multiple data to be cached and obtain the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number.

[0107] Specifically, according to an embodiment of the present invention, the data to be cached may include data of multiple business types. Accordingly, if multiple business types are involved, the data to be cached may also indicate the corresponding business types. Depending on the different business types of the data to be cached, the data formats of the multiple data to be cached may also be inconsistent.

[0108] The distributed configuration center is used to configure the number of shards corresponding to the cache, execute message notifications between the cache middleware and the local cache, etc. Before receiving multiple data to be cached, the distributed configuration center can configure the initial number of shards as the current number of shards based on historical cache information.

[0109] Furthermore, according to an embodiment of the present invention, the receiving module 301 is further configured to:

[0110] Batch receive multiple data to be cached, and perform verification processing on the multiple data to be cached;

[0111] If the verification passes, obtain the current number of shards from the distributed configuration center.

[0112] By verifying multiple cached data to filter out erroneous, duplicate, and existing cached data, it helps save cache space and improve subsequent data processing efficiency.

[0113] The sharding module 302 is configured to perform sharding processing on a plurality of data to be cached according to the current number of shards and the data serial number, and cache the plurality of data to be cached after sharding processing into the cache middleware.

[0114] Specifically, according to an embodiment of the present invention, the sharding module 302 is further configured to:

[0115] Perform a modulo operation on the data number using the value corresponding to the current number of shards as the modulo;

[0116] The shard number corresponding to each to-be-cached data is determined according to the modulo operation result, so as to implement shard processing of multiple to-be-cached data.

[0117] Through the above settings, the data numbers of multiple data to be cached are modulo the value of the current number of shards, so as to realize sharding processing of multiple data to be cached, effectively reducing the amount of data that needs to be processed subsequently and improving cache efficiency.

[0118] Furthermore, according to an embodiment of the present invention, the data to be cached further indicates a data type; and the sharding module 302 is further configured to:

[0119] The multiple cached data after sharding are cached in the cache middleware in the form of key-value pairs; wherein the key in the key-value pair includes the data type and the shard number, and the key value in the key-value pair includes the cached data stored in the corresponding shard.

[0120] Through the above settings, the data type and shard number are used as the key in the key-value pair, and the cached data in the shard is used as the corresponding value. This greatly reduces the number of keys required to set during the caching process, ensures cache performance, improves the efficiency of data caching and subsequent data queries, and is suitable for high-concurrency data caching scenarios.

[0121] Optionally, according to an embodiment of the present invention, the data type indicated by the data to be cached in the above-mentioned key is equivalent to a key prefix. If the business type is indicated in the data to be cached, the business type can also be used as a key prefix, that is, the business type and the shard number are used as the key in the key-value pair. According to a specific implementation of an embodiment of the present invention, the key prefix in the key-value pair may include the current system time or the current cache time in addition to the business type or data type. Through the above-mentioned setting, while reducing the number of keys, the key prefixes of the same batch of data to be cached are distinguished and set, so as to avoid key conflicts when further caching data in the shard in the future, ensure cache performance, and improve user experience.

[0122] Preferably, according to an embodiment of the present invention, after the step of obtaining the current number of shards from the distributed configuration center, the data processing device 300 further includes a shard number adjustment module, which is configured to:

[0123] Determine whether to expand the current number of shards based on the remaining cache space for the current shards and the amount of data to be cached.

[0124] If so, determine the actual number of shards after expansion processing based on the remaining cache space and the data volume corresponding to the multiple data to be cached, and shard the multiple data to be cached according to the actual number of shards and the data number, and cache the multiple data to be cached after sharding in the cache middleware.

[0125] Through the above settings, combined with the actual amount of data to be cached, the number of current shards and the remaining cache space, the number of shards in the cache middleware can be dynamically adjusted, further improving data processing efficiency and expanding the application scenarios of data processing.

[0126] Exemplarily, according to an embodiment of the present invention, the data processing device 300 further includes an updating module, configured to:

[0127] Using the value corresponding to the actual number of shards after the expansion process as the modulus, perform a modulo operation on the data number of the cached data in the cache middleware, and determine the actual shard number corresponding to the cached data based on the modulo operation result to update the cached data;

[0128] And send the updated results to the local cache.

[0129] To ensure the consistency of the local cache and the cached data in the cache middleware, it is necessary to synchronize and update the local cache in a timely manner after the cached data and the key corresponding to the cached data in the cache middleware are successfully changed.

[0130] The data processing module 303 is used to call the interface of the distributed configuration center to send a cache notification to the local cache after the cache middleware successfully caches the data, so as to synchronize the cached data in the cache middleware to the local cache; wherein the cached data in the local cache is a resident memory object.

[0131] Through the above settings, the distributed configuration center replaces the existing message middleware, realizing message notifications between the cache middleware and the local cache. This avoids the need for the message middleware to adjust the original system architecture, which increases the complexity of the system architecture. It also provides a second-level response to message notifications, allowing the local cache to obtain the latest cache for synchronization after the cache middleware successfully caches. At the same time, because the cached data in the local cache is a resident memory object, it avoids the situation where the business is unavailable when the local cache fails, ensuring cache performance and expanding the application scenarios of data processing.

[0132] According to one specific implementation of an embodiment of the present invention, the local cache is a cache inside the JVM (Java Virtual Machine), and the cached data in the local cache is set as a strong reference object, so that even if the current memory is insufficient, the cached data will not be recycled, ensuring that the cached data in the local cache remains resident. According to another specific implementation of an embodiment of the present invention, it is also possible to divide the local cache into multiple partitions and place the cached data in a specific partition to ensure that the cached data remains resident, so as to avoid the situation where the service is unavailable when the local cache fails.

[0133] Furthermore, according to an embodiment of the present invention, the data processing device 300 further includes a query module configured to:

[0134] Receive a target data query request, and determine a shard number corresponding to the target data according to the data number of the target data indicated by the target data query request;

[0135] The local cache is queried based on the data number and shard number, and the query result is fed back to the requester of the target data query request.

[0136] Through the above arrangement, in response to a query request, the query result can be determined quickly and accurately based on the data number.

[0137] According to the technical solution of the embodiment of the present invention, because it adopts the method of receiving multiple data to be cached, obtaining the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number; sharding the multiple data to be cached according to the current number of shards and the data number, and caching the multiple data to be cached after sharding into the cache middleware; after the cache middleware caches successfully, calling the interface of the distributed configuration center to send a cache notification to the local cache, so as to synchronously cache the data cached in the cache middleware into the local cache; wherein the cached data in the local cache is a technical means of resident memory objects, so it overcomes the high complexity of cache maintenance existing in the related art. , when the local cache fails, it will lead to business unavailability, high complexity of system architecture, low data processing efficiency, poor user experience, and it is difficult to apply to high-concurrency data caching scenarios. Technical problems, thereby achieving the ability to first use the cache middleware to store the cached data in shards, and then use the local cache to set the cached data as a resident memory object, avoiding the situation where the business is unavailable when the local cache fails. At the same time, the message notification between the cache middleware and the local cache is carried out through the distributed configuration center, which can be applied to high-concurrency data caching scenarios, improves data processing efficiency, ensures cache performance, expands the application scenarios of data processing, and improves the technical effect of user experience.

[0138] Figure 4 An exemplary system architecture 400 is shown to which the data processing method or data processing apparatus according to the embodiment of the present invention can be applied.

[0139] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0140] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Terminal devices 401, 402, and 403 can be installed with various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0141] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0142] Server 405 may be a server that provides various services, such as a server that processes data received by users using terminal devices 401, 402, and 403 (for example only). The server may analyze and process received data such as product information query requests, and feed back processing results (such as target push information and product information - for example only) to the terminal device.

[0143] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the data processing device is generally provided in the server 405 .

[0144] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0145] Reference below Figure 5 , which shows a schematic structural diagram of a computer system 500 of a terminal device or server suitable for implementing an embodiment of the present invention. Figure 5 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0146] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the system 500 are also stored in the RAM 503. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0147] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 508 as needed.

[0148] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from a removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are performed.

[0149] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a receiving module, a sharding module, and a data processing module. The names of these modules do not, in some cases, constitute a limitation on the modules themselves. For example, the receiving module may also be described as "a module for receiving a plurality of data to be cached and obtaining the current number of shards from a distributed configuration center; wherein the data to be cached indicates a data number."

[0152] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: receiving multiple data to be cached, obtaining the current number of shards from a distributed configuration center; wherein the data to be cached indicates the data number; sharding the multiple data to be cached according to the current number of shards and the data number, and caching the multiple data to be cached after sharding into the cache middleware; after the cache middleware caches successfully, calling the interface of the distributed configuration center to send a cache notification to the local cache, so as to synchronously cache the data cached in the cache middleware into the local cache; wherein the cached data in the local cache is a resident memory object.

[0153] According to the technical solution of the embodiment of the present invention, because it adopts the method of receiving multiple data to be cached, obtaining the current number of shards from the distributed configuration center; wherein the data to be cached indicates the data number; sharding the multiple data to be cached according to the current number of shards and the data number, and caching the multiple data to be cached after sharding into the cache middleware; after the cache middleware caches successfully, calling the interface of the distributed configuration center to send a cache notification to the local cache, so as to synchronously cache the data cached in the cache middleware into the local cache; wherein the cached data in the local cache is a technical means of resident memory objects, so it overcomes the problems of high complexity of cache maintenance and local When the cache fails, it will lead to business unavailability, high complexity of the system architecture, low data processing efficiency, poor user experience, and difficulty in applying to high-concurrency data caching scenarios. Technical problems can be solved by first using the cache middleware to store the cached data in shards, and then using the local cache to permanently store the cached data through strong references, avoiding the situation where business is unavailable when the local cache fails. At the same time, message notifications between the cache middleware and the local cache are carried out through the distributed configuration center, which can be applied to high-concurrency data caching scenarios, improve data processing efficiency, ensure cache performance, expand the application scenarios of data processing, and enhance the technical effect of user experience.

[0154] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Receive multiple data to be cached, and obtain the current number of shards from a distributed configuration center; wherein the data to be cached indicates a data number; Slice the plurality of data to be cached according to the current number of slicings and the data serial number, and cache the plurality of data to be cached after the slicing process in the cache middleware; After the cache middleware caches successfully, the interface of the distributed configuration center is called to send a cache notification to the local cache to synchronously cache the data cached in the cache middleware to the local cache; wherein the cached data in the local cache is a resident memory object.

2. The data processing method according to claim 1, characterized in that: The step of performing sharding processing on the plurality of data to be cached according to the current number of shards and the data serial number comprises: Taking the value corresponding to the current number of shards as modulo, performing a modulo operation on the data number; The shard number corresponding to each to-be-cached data is determined according to the modulo operation result, so as to implement shard processing of the multiple to-be-cached data.

3. The data processing method according to claim 2, characterized in that: The data to be cached also indicates the data type; The step of caching the plurality of to-be-cached data after the sharding process into the cache middleware comprises: The multiple data to be cached after the sharding is cached in the cache middleware in the form of key-value pairs; wherein the key in the key-value pair includes the data type and the shard number, and the key value in the key-value pair includes the data to be cached stored in the corresponding shard.

4. The data processing method according to claim 1, characterized in that: After the step of obtaining the current number of shards from the distributed configuration center, the method further includes: Determine whether it is necessary to expand the number of current shards according to the current number of shards, the remaining cache space corresponding to the current shards, and the data volume corresponding to the multiple data to be cached; If so, determine the actual number of shards after the expansion processing based on the remaining cache space and the data volume corresponding to the multiple data to be cached, and shard the multiple data to be cached according to the actual number of shards and the data numbers, and cache the multiple data to be cached after sharding in the cache middleware.

5. The data processing method according to claim 4, characterized in that: The method further comprises: Taking the value corresponding to the actual number of shards after the expansion process as a modulus, a modulo operation is performed on the data number of the cached data in the cache middleware, and the actual shard number corresponding to the cached data is determined according to the modulo operation result to update the cached data; And send the update result to the local cache.

6. The data processing method according to claim 1, characterized in that: The step of receiving a plurality of data to be cached and obtaining the current number of shards from the distributed configuration center also includes: receiving a plurality of data to be cached in batches, and performing verification processing on the plurality of data to be cached; If the verification passes, the current number of shards is obtained from the distributed configuration center.

7. The data processing method according to claim 1, characterized in that: The method further comprises: Receive a target data query request, and determine a shard number corresponding to the target data according to the data number of the target data indicated by the target data query request; The local cache is queried according to the data number and the shard number, and the query result is fed back to the requester of the target data query request.

8. A data processing device, characterized in that: include: A receiving module, used for receiving a plurality of data to be cached, and obtaining the current number of shards from a distributed configuration center; wherein the data to be cached indicates a data number; A sharding module, used for sharding the plurality of data to be cached according to the current number of shards and the data serial number, and caching the plurality of data to be cached after sharding into the cache middleware; A data processing module is used to call the interface of the distributed configuration center to send a cache notification to the local cache after the cache middleware caches successfully, so as to synchronously cache the data cached in the cache middleware to the local cache; wherein the cached data in the local cache is a resident memory object.

9. An electronic device for data processing, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images