A data processing method and system

Abstract

The present invention discloses a data processing method and system, which relates to the field of computer technology. A specific embodiment of the method includes: receiving a data processing request, where the data processing request indicates the data to be processed and the corresponding processing type; according to the processing type, calling a data set for storing the data to be processed and / or a bitmap indicating the storage status of the data to be processed, so as to process the data to be processed; wherein, the data set stores the data to be processed in pages, and the bitmap segmentally indicates the storage status of the data to be processed. This embodiment realizes the rapid positioning or retrieval of the data to be processed through two data structures, namely the data set and the bitmap, and improves the processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a data processing method and system. Background Art

[0002] With the development of the Internet industry, a large amount of data has emerged. Generally, such a large amount of data is stored in an unstructured type and depends on data to quickly and effectively retrieve and obtain corresponding data. To improve the retrieval efficiency of the large amount of data, accurate positioning of the data has become the key. Therefore, how to quickly and conveniently store, access, or operate the data has become the focus of research.

[0003] Currently, there are mainly two common data storage methods: creating a form in a database to store data, and at the same time creating a database index for easy positioning of the data; storing all the data in a cache in the form of a single set.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: Although the database index improves the data retrieval speed, as the amount of data increases, the creation of the database index takes a long time, and in the case of frequent data updates, the corresponding database index may be updated frequently; the data security is low, and redundant backups require a large amount of space; due to the huge amount of data, operations such as data statistics, full-scale reading, and paged reading are difficult to handle, and it is very likely to cause the data source to crash. Summary of the Invention

[0005] In view of this, the present invention provides a data processing method and system. By simultaneously using two data structures, namely a segmented bitmap and a paged data set, the method not only ensures the security of the data, but also realizes the rapid positioning of the data, as well as the convenience and effectiveness of processing such as effective verification, statistics, full-scale or paged pulling of the data.

[0006] To achieve the above object, according to a first aspect of the present invention, there is provided a data processing method, including: receiving a data processing request, where the data processing request indicates data to be processed and a corresponding processing type; according to the processing type, calling a data set for storing the data to be processed and / or a bitmap indicating the storage state of the data to be processed, so as to process the data to be processed; where the data set stores the data to be processed in pages, and the bitmap segmentally indicates the storage state of the data to be processed.

[0007] Optionally, according to the data to be processed and the bitmap segment capacity, determine a paragraph number of the bitmap for recording the storage state of the data to be processed, and a position indicated by the bitmap segment corresponding to the paragraph number; according to the data to be processed and the data set page capacity, determine a page number of the data set for storing the data to be processed, and a position indicated by the data set page corresponding to the page number.

[0008] Optionally, when the processing type is to verify the existence of the data to be processed, a bitmap for indicating the storage state of the data to be processed is called: when the storage state is the first value, it is determined that the data to be processed exists; when the indicated storage state of the data to be processed is the second value, it is determined that the data to be processed does not exist.

[0009] Optionally, when the processing type is to add or delete the data to be processed: the data to be processed is added to the position indicated by the data set page corresponding to the page number, or the data to be processed is deleted from the position indicated by the data set page corresponding to the page number; at the position indicated by the bitmap segment corresponding to the paragraph number, the storage state of the data to be processed is set to the first value, or the storage state of the data to be processed is set to the second value.

[0010] Optionally, when the processing type is to obtain all the data to be processed, the data to be processed is obtained from the data set page by page according to the page number of the data set.

[0011] To achieve the above object, according to the second aspect of the present invention, a data processing system is provided, including: a data processing device, a data storage device; wherein, the data processing device is configured to receive a data processing request, and the data processing request indicates the data to be processed and the corresponding processing type; according to the processing type, a data set for storing the data to be processed and / or a bitmap for indicating the storage state of the data to be processed are called from the data storage device to process the data to be processed; wherein, the data set stores the data to be processed page by page, and the bitmap segment by segment indicates the storage state of the data to be processed; the data storage device is configured to store the bitmap and the data set.

[0012] Optionally, the data processing device is further configured to determine the paragraph number of the bitmap for recording the storage state of the data to be processed and the position indicated by the bitmap segment corresponding to the paragraph number according to the data to be processed and the bitmap segment capacity; determine the page number of the data set for storing the data to be processed and the position indicated by the data set page corresponding to the page number according to the data to be processed and the data set page capacity.

[0013] Optionally, the data processing device is further configured to, when the processing type is to verify the existence of the data to be processed, call a bitmap for indicating the storage state of the data to be processed: when the storage state is the first value, it is determined that the data to be processed exists; when the storage state is the second value, it is determined that the data to be processed does not exist.

[0014] Optionally, it further includes: a data service device; in the case where the processing type is to add or delete the data to be processed: the data processing device is configured to push the data to be processed and the corresponding processing type to a first message queue; the data service device is configured to call, from the data storage device, a data set for storing the data to be processed and a bitmap indicating the storage state of the data to be processed.

[0015] Optionally, the data service device is further configured to add the data to be processed to the position indicated by the data set page corresponding to the page number, or delete the data to be processed from the position indicated by the data set page corresponding to the page number; at the position indicated by the bitmap segment corresponding to the paragraph number, set the storage state of the data to be processed to a first value, or set the storage state of the data to be processed to a second value.

[0016] Optionally, in the case where the processing type is to obtain all the data to be processed: the data service device is configured to call, from the data storage device, a data set for storing the data, and page through the data set according to the page numbers of the data set to obtain the data to be processed, and push the obtained data to be processed to a second message queue; the data processing device is configured to obtain the data to be processed from the second message queue.

[0017] Optionally, the data storage device is Redis, and Redis is used to store the data set, the bitmap, the first message queue, and the second message queue.

[0018] To achieve the above object, according to the third aspect of the present invention, there is provided a server for data processing, including: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data processing methods described above.

[0019] To achieve the above object, according to the fourth aspect of the present invention, there is provided a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, it implements any one of the data processing methods described above.

[0020] The technical solution of the above invention has the following advantages or beneficial effects: Since the paged data set stores the data to be processed while using the segmented bitmap to indicate the storage status of the data to be processed, it ensures the consistency, reliability and security of the data to be processed, and also avoids the large amount of space occupied by redundant backups; at the same time, based on the segmented bitmap, fast verification and quantity statistics of the data are realized; in addition, based on the paged data set, fast and effective paging or full-volume reading of the data can also be realized.

[0021] The further effects of the above non-conventional optional methods will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Among them:

[0023] Figure 1 is a schematic diagram of the main process of the data processing method according to an embodiment of the present invention;

[0024] Figure 2a is a schematic diagram of a paged data set according to an embodiment of the present invention;

[0025] Figure 2b is a schematic diagram of a segmented bitmap according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of the main structure of a data processing system according to an embodiment of the present invention;

[0027] Figure 4 is an exemplary system architecture diagram to which an embodiment of the present invention can be applied;

[0028] Figure 5 is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] The following describes exemplary embodiments of the present invention with reference to the drawings, including various details of the embodiments of the present invention to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description below omits descriptions of well-known functions and structures.

[0030] See Figure 1 , an embodiment of the present invention provides a data processing method, which may specifically include the following steps:

[0031] Step S101: Receive a data processing request, where the data processing request indicates the data to be processed and the corresponding processing type.

[0032] In the embodiments of the present application, the data to be processed includes, but is not limited to: numerical values, data IDs, data numbers, data identifiers, etc. Among them, the data corresponding to data IDs, data numbers, data identifiers, etc. can be data of any form or content, such as pictures, texts, videos, files, numerical values, strings, etc. The processing types include, but are not limited to: verification of data validity or existence, statistics of data volume, reading, obtaining, or pulling of data, etc. Specifically, taking the data to be processed as a data ID as an example for illustration, usually data IDs are of numerical type. Facing a large amount of data IDs, the common processing operations include: verifying the validity of the data ID, using the data ID for positioning or retrieving the corresponding data, counting the total amount of data IDs, pulling all data IDs in full or in pages, etc.

[0033] Step S102: According to the processing type, call the data set for storing the data to be processed and / or the bitmap indicating the storage status of the data to be processed, so as to process the data to be processed; wherein, the data set stores the data to be processed in pages, and the bitmap segmentally indicates the storage status of the data to be processed.

[0034] To prevent problems such as low efficiency and long time consumption in data retrieval or positioning caused by an excessive number of data to be processed, it is considered to use a paged data set to store the data to be processed, and at the same time use a segmented bitmap to indicate the storage status of the data to be processed, that is, whether the data to be processed is valid or exists. Among them, the elements of the data set are the data to be processed stored in the data set, and the page capacity and the number of data set pages of the data set can be set according to actual needs to store the data to be processed in different data set pages, so as to quickly locate and process the data to be processed by positioning the data set page storing the data to be processed. It should be noted that in addition to storing the data to be processed, any data such as text, pictures, videos, files, strings, etc. corresponding to the data to be processed (such as data ID, data number, data identifier) can also be stored in the data set according to actual needs. The bitmap indicates the storage status of the data to be processed according to the value stored in each bit. For example, different storage statuses are indicated by storing different first values and second values, and the capacity of the bitmap segment and the number of bitmap segments can be set according to actual needs to use the values stored in different positions in different bitmap segments to indicate the storage status of the data to be processed, so that the storage status of the data to be processed can be quickly obtained by determining the bitmap segment corresponding to the data to be processed, and then processing such as verification and statistics of the data to be processed can be realized. It can be understood that the first value and the second value can be any values with different values. For example, the first value of 1 represents that the data to be processed exists or is valid, the second value of 0 represents that the data to be processed does not exist or is invalid, or the first value is 1 and the second value is 2, or the first value is true and the second value is false, etc.

[0035] See Figure 2a , a schematic diagram of a paged data set is provided. The data set stores the data to be processed by paging, and the page numbers corresponding to the set data set page capacity of 100 are 0, 1, 130, etc. Among them, the data set page with page number 0 can accommodate the data to be processed from 1 to 99, but the actually stored data to be processed are 1, 2, 4, 5, 6, 99, etc.; the data set page with page number 1 can store the data to be processed from 100 to 199, and the data set page with page number 130 can store the data to be processed from 1300 to 1399. That is to say, the number of data to be processed that can be stored in different data set pages is the same, and the data to be processed that can be stored is different.

[0036] See Figure 2b , a schematic diagram of a segmented bitmap is provided. Since the bitmap only indicates the storage status of the data to be processed by the value (such as 0 or 1) stored in each bit and does not store the data to be processed itself, it occupies little space and has extremely fast processing speed. Therefore, the capacity of the bitmap segment can be much larger than the capacity of the data set page, and can be in the order of hundreds, thousands, tens of thousands, hundreds of thousands, millions, etc. Specifically, Figure 2bTaking the bitmap segment capacity as 1 million, the first value of 1 indicating that the data to be processed is valid or exists, and the second value of 0 indicating that the data to be processed does not exist or is invalid as an example, where the bitmap segment with paragraph number 0 indicates the storage status of the data to be processed from 0 to 999999 through the values on each bit with position numbers from 0 to 999999, the bitmap segment with paragraph number 1 indicates the storage status of the data to be processed from 1000000 to 1999999 through the values on each bit with position numbers from 0 to 999999, and the bitmap segment with paragraph number 2 indicates the storage status of the data to be processed from 2000000 to 2999999 through the values on each bit with position numbers from 0 to 999999. Further, taking the data to be processed as 1 and 4 as examples, according to the bitmap segment capacity of 1 million, it can be known that the paragraph number of the bitmap segment recording the storage status of the data to be processed 1 and 4 is 0, and the corresponding position numbers in the bitmap segment are 1 and 4. And according to the values stored at the corresponding positions, the storage status of the data to be processed 1 is 0, that is, the data to be processed 1 does not exist or is invalid, and the storage status of the data to be processed 4 is 1, that is, the data to be processed 4 exists or is valid.

[0037] Since there are a large number of segmented bitmaps and paged data sets, in order to be able to quickly process the data to be processed, it is necessary to more quickly and effectively find or locate the data set page or bitmap segment corresponding to the data to be processed according to the correspondence between the data to be processed and the data set page or bitmap segment.

[0038] In an optional manner, according to the data to be processed and the bitmap segment capacity, determine the paragraph number of the bitmap recording the storage status of the data to be processed, and the position indicated by the bitmap segment corresponding to the paragraph number; according to the data to be processed and the data set page capacity, determine the page number of the data set storing the data to be processed, and the position indicated by the data set page corresponding to the page number.

[0039] Specifically, still referring to Figure 2b , taking the data to be processed as 1301 as an example, since the bitmap segment capacity is 1 million, then by calculating the quotient and remainder of the data to be processed and the bitmap segment capacity, it can be known that the quotient of the data to be processed 1301 and the bitmap segment capacity is 0 and the remainder is 1301, that is, the paragraph number of the bitmap segment indicating the storage status of the data to be processed 1301 is 0, and the value stored at the bit numbered 1301 in the bitmap segment with paragraph number 0 indicates the storage status of the data to be processed 1301;

[0040] Specifically, still referring to Figure 2a, still taking the data to be processed as 1301 as an example for illustration. Since the capacity of the data set page is 100, by calculating the quotient and remainder of the data to be processed and the capacity of the data set page, it can be known that the quotient of the data to be processed 1301 and the capacity of the data set page 100 is 13, and the remainder is 1. That is, the page number of the data set page used to store the data to be processed 1301 is 13, and the storage order or the corresponding position number of the data to be processed 1301 in the data set page with page number 130 is 1.

[0041] It can be understood that since the storage structures of the data to be processed of the segmented bitmap and the paged data set are different, the corresponding stored contents are also different. Therefore, for different processing types, such as statistics, verification, full - volume acquisition, or paged acquisition of the data to be processed, the processing efficiencies are different. For example, the segmented bitmap is more suitable for counting the number of data to be processed or verifying the existence of the data to be processed. However, at the same time, since the segmented bitmap does not store the data to be processed itself, it is not suitable for reading the data to be processed, while the paged data set is suitable for paged reading or obtaining the data to be processed. Therefore, for different processing types, the segmented bitmap or / and the data set can be selected and called according to actual needs to implement the processing of the data to be processed. At the same time, to ensure the consistency between the data to be processed actually stored in the data set page and the storage status of the data to be processed indicated by the segmented bitmap, the consistency between the two can be verified irregularly or periodically according to actual needs, so as to ensure the reliability and security of the data and avoid the occupation of a large amount of space by data backup.

[0042] In an optional implementation manner, when the processing type is to verify whether the data to be processed exists, the bitmap used to indicate the storage status of the data to be processed is called: when the storage status is the first value, it is determined that the data to be processed exists; when the indicated storage status of the data to be processed is the second value, it is determined that the data to be processed does not exist.

[0043] Specifically, still referring to Figure 2b , taking the verification of whether the data to be processed 4 exists as an example for illustration: First, according to the quotient and remainder of the data to be processed 4 and the bitmap segment capacity of 1 million, it can be known that the paragraph number of the bitmap segment recording the data to be processed 4 is 0, and the value stored in the bit corresponding to the position number 4 in paragraph number 0 indicates the storage status of the data to be processed; the bitmap is called to obtain the value stored in the 4th bit of bitmap segment 0. According to the stored value 1, it is determined that the data to be processed 4 exists.

[0044] Similarly, when calling the bitmap to determine whether the data to be processed exists or is valid, the quantity or total quantity of the data to be processed can be statistically calculated based on the storage status corresponding to the data to be processed obtained from the bitmap.

[0045] In an alternative embodiment, when the processing type is to add or delete the data to be processed: add the data to be processed to the position indicated by the data set page corresponding to the page number, or delete the data to be processed from the position indicated by the data set page corresponding to the page number; at the position indicated by the bitmap segment corresponding to the paragraph number, set the storage state of the data to be processed to a first value, or set the storage state of the data to be processed to a second value.

[0046] Still referring to Figure 2a and Figure 2b , taking the addition of the data to be processed 3 as an example for illustration, according to the quotient 0 of the data to be processed 3 and the data set page capacity 100, it can be known that the page number of the data set for storing the data to be processed 3 is 0, and the data to be processed 3 is added to the data set page. Further, for the convenience of processing the data to be processed, the remainder 3 of the data to be processed 3 and the data set page capacity 100 can be considered to determine the storage order or position number of the data to be processed 3 in the data set page. At the same time, since the data to be processed 3 is added to the data set page, the storage state of the data to be processed 3 in the bitmap needs to be changed accordingly. According to the quotient and remainder of the data to be processed 3 and the bitmap segment capacity of 1 million, it can be known that the storage state of the data to be processed 3 is recorded at the bit with the paragraph number 0 and the position number 3. Accordingly, the data at this bit is set to the first value (such as 1) to indicate that the data to be processed 3 already exists or is valid.

[0047] Still referring to Figure 2a and Figure 2b , taking the deletion of the data to be processed 4 as an example for illustration, according to the quotient 0 of the data to be processed 4 and the data set page capacity 100, it can be known that the page number of the data set for storing the data to be processed 4 is 0, and the data to be processed 4 is deleted from this data set page. Further, for the convenience of processing the data to be processed, the remainder 4 of the data to be processed 4 and the data set page capacity 100 can be considered to determine the storage order or position number of the data to be processed 4 in the data set page. At the same time, since the data to be processed 4 is deleted from the data set page, the storage state of the data to be processed 4 in the bitmap needs to be changed accordingly. According to the quotient and remainder of the data to be processed 4 and the bitmap segment capacity of 1 million, it can be known that the storage state of the data to be processed 4 is recorded at the bit with the paragraph number 0 and the position number 4. Accordingly, the data at this bit is set to the second value (such as 0) to indicate that the data to be processed 4 no longer exists or is invalid.

[0048] It can be understood that there are numerous data processing requests, and different data processing requests may involve the same processing of the same data to be processed, such as both deleting the data to be processed 4 and adding the data to be processed 3. To avoid wasting resources, according to actual needs, the status of the data to be processed can be determined from the segmented bitmap first, and then it can be judged whether the data to be processed has been deleted, added, etc. In this way, the waste of resources and time caused by calling the data set again to add or delete the data to be processed is avoided.

[0049] In an optional implementation manner, when the processing type is to obtain all the data to be processed, the data to be processed is obtained from the data set page by page according to the page number of the data set.

[0050] See Figure 2a , when obtaining all the data to be processed, the data set is called to obtain the data to be processed. At the same time, since the amount of data to be processed is extremely large, it takes a long time to pull all of them at once and the operation rate is low. Based on this, the corresponding data to be processed is obtained from the data set page by page. Specifically, according to the page numbers 0, 1, …… 13, etc. of the data set pages, the data to be processed 1, 2, 4, 5, 6, ……, 99, etc. are first obtained from the data set page with page number 0; then the data to be processed 101, 102, 104, etc. are obtained from the data set page with page number 1 until all the data to be processed in all the data set pages are obtained. It can be understood that in the actual application process, the amount of data obtained at one time depends on the data set page capacity. Therefore, the speed of reading or pulling the data to be processed can be adjusted by means of the data set page capacity.

[0051] Based on the data processing method provided in the above embodiments, by using two data structures, namely the segmented bitmap and the paged data set, to store data, both the security of the data is ensured, and the rapid positioning of the data is achieved, as well as the convenience and effectiveness of the effective verification, statistics, full - volume or paged pulling of the data.

[0052] See Figure 3 , on the basis of the above embodiments, an embodiment of the present invention provides a data processing system 300, including: a data processing device 301 and a data storage device 302; wherein, the data processing device 301 is used to receive a data processing request, and the data processing request indicates the data to be processed and the corresponding processing type; according to the processing type, call the data set for storing the data to be processed and / or the bitmap indicating the storage status of the data to be processed from the data storage device 302 to process the data to be processed; wherein, the data set stores the data to be processed page by page, and the bitmap indicates the storage status of the data to be processed segment by segment; the data storage device 302 is used to store the bitmap and the data set.

[0053] Specifically, the data processing device 301 can be deployed either on the user APP or the user client, or can be deployed independently to receive data processing requests initiated by the user. The data storage device can be a device with storage space such as memory, cache, Redis, etc., and interacts or communicates with the data processing device 301.

[0054] In an alternative embodiment, the data processing device 301 is further configured to determine, according to the data to be processed and the bitmap segment capacity, the paragraph number of the bitmap recording the storage state of the data to be processed, and the position indicated by the bitmap segment corresponding to the paragraph number; and determine, according to the data to be processed and the data set page capacity, the page number of the data set storing the data to be processed, and the position indicated by the data set page corresponding to the page number.

[0055] In an alternative embodiment, the data processing device 301 is further configured to, when the processing type is to verify whether the data to be processed exists, call the bitmap indicating the storage state of the data to be processed: when the storage state is the first value, determine that the data to be processed exists; when the storage state is the second value, determine that the data to be processed does not exist.

[0056] In an alternative embodiment, the data processing system 300 further includes: a data service device 303; when the processing type is to add or delete the data to be processed: the data processing device 301 is configured to push the data to be processed and the corresponding processing type to a first message queue; the data service device 302 is configured to call, from the data storage device 302, the data set for storing the data to be processed and the bitmap indicating the storage state of the data to be processed.

[0057] The data service device 303 can be independently deployed separate from the data processing device 301, such as on other servers, etc. It can both communicate with the data processing device 301 and communicate with the data storage device 302. It can be understood that when the received data processing type is to add or delete the data to be processed, the data processing device 301 can either directly call the data set and bitmap in the data storage device 302 to add or delete the data to be processed, or call the data service device 303, and through the data service device, call the data set and bitmap in the data storage device 302 to add or delete the data to be processed. It should be noted that when the data processing device 301 calls the data service device 303, the data processing device 301 realizes communication with the data service device 303 through the first message queue. That is to say, when the data processing device 301 receives a data processing request to add or delete the data to be processed, the data processing device 301 calls the first message queue from the data storage device 302 or the message middleware, and pushes the data to be processed and the processing type indicated by the data processing request to the first message queue, so that the data service device 303 directly calls the data set and bitmap in the data storage device 302 according to the data to be processed and the processing type received from the first message queue, for adding or deleting the data to be processed in the data set and bitmap respectively. In this way, decoupling between the data processing device 301 and the data service device 303 is achieved through the first message queue, relieving the pressure on the data processing device 301 or the data service device 303 to process data processing requests. At the same time, through the real-time monitoring of the first message queue at the data service device 303, loss of the data to be processed and the processing type is prevented.

[0058] In an alternative embodiment, the data service device 303 is further configured to add the data to be processed to the position indicated by the data set page corresponding to the page number, or delete the data to be processed from the position indicated by the data set page corresponding to the page number; at the position indicated by the bitmap segment corresponding to the paragraph number, set the storage state of the data to be processed to the first value, or set the storage state of the data to be processed to the second value.

[0059] In an alternative embodiment, when the processing type is to obtain all the data to be processed: the data service device 303 is configured to call the data set for storing the data from the data storage device 302, page through the data set according to the page numbers of the data set to obtain the data to be processed, and push the obtained data to be processed to the second message queue; the data processing device 301 is configured to obtain the data to be processed from the second message queue.

[0060] It can be understood that when the type of data processing received is to obtain all data to be processed, the data processing device 301 can either directly call the data set in the data storage device 302 to read or pull the data to be processed, or call the data service device 303, and the data service device calls the data set in the data storage device 302. It should be noted that when the data processing device 301 calls the data service device 303, the data processing device 301 obtains the data to be processed through the second message queue. The second message queue can be either a message queue that has been established and stored in the data storage device 302 or the message middleware, or a message queue newly established by the data processing device 301 directly in the data storage device or the message middleware. Specifically, taking the data processing device 301 creating a new second message queue as an example, the data processing device 301 calls the data service device with the identifier of the second message queue, so that the data service device 303 directly calls the data set in the data storage device 302, pulls the data to be processed by paging through the page number of the data combination page, and pushes the paged data to be processed to the second message queue, so that the data processing device 301 can obtain the data to be processed from the second message queue. In this way, a message producer and consumer mode is formed between the data processing device 301 and the data service device 303. At the same time, the message production or consumption pressure between the data processing device 301 and the data service device 303 can be balanced through the message queue blocking mode or the length of the message queue. For example, when the message consumption volume of the data processing device 301 is less than the message production volume of the data service device 303, the data pulling ability of the data service device 303 for the data to be processed can be reduced by reducing the length of the second message queue. When the message consumption volume of the data processing device 301 is greater than the message production volume of the data service device 303, the data pulling ability of the data service device 303 for the data to be processed can be improved by increasing the number of threads of the data service device 303.

[0061] In an alternative embodiment, the data storage device 302 is Redis, and Redis is used to store the data set, the bitmap, the first message queue, and the second message queue.

[0062] It can be understood that when the data storage device is Redis, it can be used as a cache to store the data set and the bitmap, and can also store the message queue. Or, according to actual requirements, a message middleware can be used to store the message queue.

[0063] Figure 4 An exemplary system architecture 400 to which the data processing method or data processing device of the embodiments of the present invention can be applied is shown.

[0064] As Figure 4As shown, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 is used to provide a medium for communication links between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0065] Users can use the terminal devices 401, 402, 403 to interact with the server 405 through the network 404 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] The terminal devices 401, 402, 403 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0067] The server 405 may be a server providing various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 401, 402, 403. The background management server can analyze and process data such as product information query requests received, and feedback the processing results (such as the storage status of data to be processed) to the terminal devices.

[0068] It should be noted that the data processing method provided by the embodiments of the present invention is generally executed by the server 405.

[0069] It should be understood that Figure 4 the numbers of terminal devices, networks, and servers in

[0070] are merely illustrative. According to the implementation requirements, there may be any number of terminal devices, networks, and servers.

[0070] Next, refer to Figure 5 which shows a schematic structural diagram of a computer system 500 of a terminal device suitable for implementing the embodiments of the present invention. Figure 5 The terminal device shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0071] As Figure 5As shown, computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage section 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0072] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom can be installed into the storage section 508 as needed.

[0073] Specifically, according to an embodiment disclosed by the present invention, the processes described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment disclosed by the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from the removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, the above functions defined in the system of the present invention are executed.

[0074] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0076] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist separately without being assembled into the device. The above computer-readable medium carries one or more programs. When the one or more programs are executed by the device, the device includes: receiving a data processing request, where the data processing request indicates the data to be processed and the corresponding processing type; according to the processing type, calling a data set for storing the data to be processed and / or a bitmap indicating the storage state of the data to be processed, so as to process the data to be processed; wherein, the data set stores the data to be processed in pages, and the bitmap indicates the storage state of the data to be processed in segments.

[0077] According to the technical solution of the embodiment of the present invention, the technical solution of the above invention has the following advantages or beneficial effects: Since the paged data set stores the data to be processed while using the segmented bitmap to indicate the storage state of the data to be processed, it ensures the consistency, reliability and security of the data to be processed, and avoids the large amount of space occupied by redundant backups; at the same time, based on the segmented bitmap, fast verification and quantity statistics of the data are realized; in addition, fast and effective paging or full-volume reading of the data can also be realized based on the paged data set.

[0078] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, including: Receiving a data processing request, where the data processing request indicates the data to be processed and the corresponding processing type; According to the processing type, calling a data set for storing the data to be processed and / or a bitmap indicating the storage status of the data to be processed, for processing the data to be processed; wherein, the data set stores the data to be processed in pages, and the bitmap indicates the storage status of the data to be processed in segments; the number of data to be processed that can be stored in different data set pages is the same, and the data to be processed that can be stored is different; According to the data to be processed and the bitmap segment capacity, determining the paragraph number of the bitmap recording the storage status of the data to be processed, and the position indicated by the bitmap segment corresponding to the paragraph number; According to the data to be processed and the data set page capacity, determining the page number of the data set storing the data to be processed, and the position indicated by the data set page corresponding to the page number.

2. The data processing method according to claim 1, characterized in that, When the processing type is to verify whether the data to be processed exists, calling a bitmap indicating the storage status of the data to be processed: When the storage status is the first value, determining that the data to be processed exists; when the indicated storage status of the data to be processed is the second value, determining that the data to be processed does not exist.

3. The data processing method according to claim 1, characterized in that, When the processing type is to add or delete the data to be processed: Adding the data to be processed to the position indicated by the data set page corresponding to the page number, or deleting the data to be processed from the position indicated by the data set page corresponding to the page number; At the position indicated by the bitmap segment corresponding to the paragraph number, setting the storage status of the data to be processed to the first value, or setting the storage status of the data to be processed to the second value.

4. The data processing method according to claim 1, characterized in that, When the processing type is to obtain all the data to be processed, obtaining the data to be processed from the data set page by page according to the page number of the data set.

5. A data processing system, characterized in that, including: A data processing device, a data storage device; wherein, The data processing device is used to receive a data processing request, where the data processing request indicates the data to be processed and the corresponding processing type; according to the processing type, calling a data set for storing the data to be processed and / or a bitmap indicating the storage status of the data to be processed from the data storage device, for processing the data to be processed; wherein, the data set stores the data to be processed in pages, and the bitmap indicates the storage status of the data to be processed in segments; the number of data to be processed that can be stored in different data set pages is the same, and the data to be processed that can be stored is different; The data storage device is used to store the bitmap, the data set; The data processing device is further used to, According to the data to be processed and the bitmap segment capacity, determining the paragraph number of the bitmap recording the storage status of the data to be processed, and the position indicated by the bitmap segment corresponding to the paragraph number; According to the data to be processed and the data set page capacity, determining the page number of the data set storing the data to be processed, and the position indicated by the data set page corresponding to the page number.

6. The data processing system according to claim 5, characterized in that, The data processing device is further configured to when the processing type is to verify the existence of the data to be processed, call a bitmap for indicating the storage status of the data to be processed: when the storage status is a first value, determine that the data to be processed exists; when the storage status is a second value, determine that the data to be processed does not exist.

7. The data processing system according to claim 5, characterized in that, It further includes: a data service device; when the processing type is to add or delete the data to be processed: the data processing device is configured to push the data to be processed and the corresponding processing type to a first message queue; the data service device is configured to call, from the data storage device, a data set for storing the data to be processed and a bitmap for indicating the storage status of the data to be processed.

8. The data processing system according to claim 7, characterized in that, The data service device is further configured to add the data to be processed to the position indicated by the data set page corresponding to the page number, or delete the data to be processed from the position indicated by the data set page corresponding to the page number; at the position indicated by the bitmap segment corresponding to the paragraph number, set the storage status of the data to be processed to the first value, or set the storage status of the data to be processed to the second value.

9. The data processing system according to claim 7, characterized in that, when the processing type is to obtain all the data to be processed: the data service device is configured to call, from the data storage device, a data set for storing the data, and page through the data set to obtain the data to be processed according to the page numbers of the data set, and push the obtained data to be processed to a second message queue; the data processing device is configured to obtain the data to be processed from the second message queue.

10. The data processing system according to claim 9, wherein, The data storage device is Redis, and Redis is used to store the data set, the bitmap, the first message queue, and the second message queue.

11. A server for data processing, characterized in that, It includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-4.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1-4.

Images