A data processing method, device, system, electronic device, and storage medium

Abstract

The present disclosure provides a data processing method, device, system, electronic device and storage medium, relates to the field of data processing, and particularly relates to the field of big data technology. The specific implementation scheme is: receiving resource content data sent by an RCMC receiving module, the NID of the resource content data having a mapping relationship with an RCMC processing module; constructing a mutex for the NID of the received resource content data, and sequentially writing the resource content data carrying the same NID into a database based on the mutex; after completing the writing of one piece of resource content data, adding the resource content data to a local queue; based on the mutex of the NID of each piece of resource content data in the local queue, pushing the resource content data in the local queue to a first message middleware corresponding to the local queue, so that a cache service acquires and caches the resource content data from the first message middleware. The cache service can avoid caching an error version of the resource content data.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the field of big data technology. Background Technology

[0002] In a distributed caching system, when the receiving module receives updated resource content data, it can write the updated resource content data to a message broker. The writing module can then retrieve the updated resource content data from this message broker, write the retrieved resource content data to the database, and write the retrieved resource content data to another message broker for the caching layer to retrieve the updated resource content data. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, system, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a data processing method is provided, the method being applied to the RCMC processing module of a resource content management center, the method comprising:

[0005] The system receives resource content data sent by the RCMC receiving module. The resource content data carries a message identifier (NID), and the NID of the resource content data has a mapping relationship with the RCMC processing module.

[0006] Construct a mutex lock for the NID of the received resource content data, and write the resource content data with the same NID into the database sequentially based on the mutex lock;

[0007] After each piece of resource content data is written, the resource content data is added to the local queue;

[0008] Based on the mutex lock of the NID of each resource content data in the local queue, the resource content data in the local queue is pushed to the first message middleware corresponding to the local queue, so that the caching service can obtain and cache the resource content data from the first message middleware.

[0009] According to a second aspect of this disclosure, a data processing apparatus is provided, the apparatus being applied to a Resource Content Management Center (RCMC) processing module, the apparatus comprising:

[0010] The receiving unit is used to receive resource content data sent by the RCMC receiving module. The resource content data carries a message identifier (NID), and the NID of the resource content data has a mapping relationship with the RCMC processing module.

[0011] The write unit is used to construct a mutex lock for the NID of the received resource content data, and write the resource content data with the same NID into the database sequentially based on the mutex lock;

[0012] The add unit is used to add the resource content data to the local queue after each write is completed;

[0013] The push unit is used to push the resource content data in the local queue to the first message middleware corresponding to the local queue based on the mutex lock of the NID of each resource content data in the local queue, so that the caching service can obtain and cache the resource content data from the first message middleware.

[0014] According to a third aspect of this disclosure, a data processing system is provided, comprising: multiple RCMC receiving modules, multiple RCMC processing modules, a database, a first message middleware, and a caching service;

[0015] Each RCMC receiving module is used to send resource content data to the RCMC processing module that has a mapping relationship with the NID of the resource content data;

[0016] Each RCMC processing module is used to execute the steps of the method described in the first aspect above;

[0017] The caching service is used to obtain and cache resource content data from the first message middleware.

[0018] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0022] According to a fifth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.

[0023] According to a sixth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0026] Figure 1 A schematic diagram of a writing module provided in an embodiment of this disclosure;

[0027] Figure 2 A flowchart of a data processing method provided in an embodiment of this disclosure;

[0028] Figure 3 An exemplary schematic diagram illustrating a method for an RCMC processing module to write data to a database, as provided in an embodiment of this disclosure;

[0029] Figure 4 This is an exemplary schematic diagram illustrating a method for an RCMC receiving module to send data to an RCMC processing module, as provided in an embodiment of this disclosure.

[0030] Figure 5 This is an exemplary schematic diagram illustrating a method for an RCMC processing module to push data to a first message middleware, as provided in an embodiment of this disclosure.

[0031] Figure 6 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this disclosure;

[0032] Figure 7 A block diagram of a data processing system provided in an embodiment of this disclosure;

[0033] Figure 8 This is a block diagram of an electronic device used to implement the data processing method of the embodiments of this disclosure. Detailed Implementation

[0034] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] In related technologies, after the writing module obtains resource content data from a message middleware, it can write the obtained resource content data into a database, and then write the resource content data already written into the database into another message middleware, so that the caching service can obtain and cache the resource content data from the message middleware.

[0036] Since resource content data may not carry a version number, when the resource content data of the same resource is updated frequently, or when the writing module experiences machine jitter or other failures, the order in which the resource content data is written to another message middleware may be inconsistent with the actual update order of the resource content data, resulting in the caching service caching the wrong version of the resource content data.

[0037] For example, if the resource content data of the same resource is frequently updated, assuming the resource is an article, and the article is first modified by the author and then immediately blocked by the administrator, then the writing module can obtain two pieces of resource content data: the first is the data on the author's modification of the article, and the second is the data on the article being blocked by the administrator.

[0038] The write module first writes the first piece of data to the database, and then writes the second piece of data to the database. Because the second piece of data is smaller, it's possible that the second piece of data is processed and pushed to the message middleware first, while the first piece of data is processed and pushed to the message middleware later. This causes the caching service to retrieve the second piece of data first, and then the first piece of data. Consequently, the caching service overwrites the first piece of data with the cached second piece of data, resulting in a mismatch between the cached data version and the data version stored in the database, leading to an error in the cached data version. Furthermore, after the business service retrieves the second piece of data from the caching service, it will display the article as if it were edited by the author, causing the article to not be blocked and resulting in a business error.

[0039] To address the aforementioned problems, this disclosure provides a data processing method applied to a Resource Content Management Center (RCMC) processing module. This RCMC processing module can be any RCMC processing module within a data processing system. The data processing system includes multiple RCMC receiving modules, multiple RCMC processing modules, a database, a first message middleware, and a caching service.

[0040] like Figure 1 As shown, in this embodiment of the disclosure, the writing module in the prior art is split into an RCMC receiving module and an RCMC processing module. Figure 1The example shows three RCMC receiving modules and three RCMC processing modules. In actual implementation, the number of RCMC receiving modules and RCMC processing modules included in the data processing system is not limited to this.

[0041] The RCMC receiving module receives resource traffic and sends it to the RCMC processing module. Resource traffic includes multiple pieces of resource content data. This resource content data can be generated from user-posted articles, videos, and comments; it can also be generated from user actions such as modifying, forwarding, or liking published content; and it can be generated from administrator actions such as blocking or deleting user-posted content.

[0042] The RCMC processing module is used to write the received resource content data into the database and then push it to the first message middleware, so that the caching service can obtain the resource content data from the first message middleware and cache it in the caching layer.

[0043] In this embodiment of the disclosure, the RCMC receiving module can be a Pre-RCMC, and the RCMC processing module can be an RCMC instance.

[0044] The data processing method provided in the embodiments of this disclosure will be described in detail below.

[0045] like Figure 2 As shown, the data processing method provided in this embodiment includes:

[0046] S201. Receive resource content data sent by the RCMC receiving module. The resource content data carries a message identity (NID), and the NID of the resource content data has a mapping relationship with the RCMC processing module.

[0047] All resource content data for the same resource carry the same NID.

[0048] In this embodiment of the disclosure, the NID of resource content data and the RCMC processing module have a preset mapping relationship, and thus, resource content data with the same NID can be mapped to the same RCMC processing module.

[0049] S202. Construct a mutex lock for the NID of the received resource content data, and write the resource content data with the same NID into the database sequentially based on the mutex lock.

[0050] Optionally, the RCMC processing module has a Licache, and embodiments of this disclosure can construct mutex locks for NIDs through the Licache. The Licache is an externally used cache that has the function of constructing mutex locks.

[0051] By constructing a mutex lock for the NID of the received resource content data, it can be guaranteed that only one thread can process resource content data with that NID at any given time. In other words, at any given moment, the RCMC processing module can only process one of multiple resource content data entries with the same NID. Only after completing the writing of one resource content data entry can it continue processing the next resource content data entry with the same NID. This ensures that multiple resource content data entries with the same NID are written to the database in the order they are received.

[0052] As an example, such as Figure 3 As shown, RCMC processing module 1 sequentially receives three resource content data belonging to the same resource, referred to as resource 1”, resource 1', and resource 1 respectively. Because these three resource content data belong to the same resource, they have the same NID.

[0053] RCMC processing module 1 first receives resource 1”, then constructs a mutex lock for the NID of resource 1”, synchronizes resource 1” to database read / write module 1, and then writes resource 1” to the database through database read / write module 1. During this process, RCMC processing module 1 receives resource 1’. Because the NID of resource 1’ and the NID of resource 1” have the same mutex lock, RCMC processing module 1 can only write resource 1’ to the database after resource 1” has been written to the database. Similarly, RCMC processing module 1 can only write resource 1 to the database after writing resource 1’ to the database, avoiding order disorder.

[0054] Similarly, RCMC processing module 2 can use a mutex lock based on the NIDs of resource 2”, resource 2’, and resource 2 to sequentially write resource 2”, resource 2’, and resource 2 into the database via database read / write module 2. RCMC processing module 3 can use a mutex lock based on the NIDs of resource 3”, resource 3’, and resource 3 to sequentially write resource 3”, resource 3’, and resource 3 into the database via database read / write module 3.

[0055] Optionally, the database read / write module in this embodiment of the disclosure can be a global content manager service (GCMS).

[0056] S203. After each piece of resource content data is written, add the resource content data to the local queue.

[0057] S204. Based on the mutex lock of the NID of each resource content data in the local queue, push the resource content data in the local queue to the first message middleware corresponding to the local queue, so that the caching service can obtain and cache the resource content data from the first message middleware.

[0058] In this embodiment of the disclosure, the RCMC processing module can generate a local queue for each first message middleware, and resource content data that needs to be pushed to the same message middleware can be added to the same local queue.

[0059] Using the above technical solution, the RCMC processing module can receive resource content data corresponding to NIDs that have a mapping relationship with itself. This allows resource content data with the same NID to be received by the same RCMC processing module. Based on a mutex lock for NIDs, the RCMC processing module sequentially writes resource content data carrying the same NID into the database. Due to the restriction of the mutex lock, the RCMC processing module can only process one piece of resource content data with the same NID at a time, preventing the later-updated resource content data from being written first, thus ensuring that resource content data for the same resource is written sequentially to the database and local queue. Because the RCMC processing module also utilizes the NID mutex lock when pushing resource content data from the local queue to the first message middleware, it ensures that resource content data for the same resource is pushed sequentially to the first message middleware. This allows the caching service to obtain and cache resource content data for the same resource in the correct order, avoiding inconsistencies between the data version in the caching layer and the database, and ensuring that the caching layer can cache the correct version of resource content data.

[0060] In addition, if the data version cached by the caching layer is incorrect, the only solution is to manually update the data cached by the caching layer. If the error is not detected and manually updated in time, the cached data will be incorrect for a long time. The embodiments of this disclosure avoid the problem of time sequence disorder of resource content data obtained by the caching service, and can avoid the caching layer from caching incorrect data for a long time.

[0061] In this embodiment of the disclosure, the resource content data obtained by the RCMC receiving module comes from the second message middleware, which is used to cache the updated resource content data.

[0062] When updated resource content data exists, the updated resource content data will be published to the second message middleware. In this embodiment of the present disclosure, multiple RCMC receiving modules can act as consumers to consume the resource content data in the second message middleware. The same resource content data in the second message middleware can only be consumed by one RCMC receiving module. When a resource content data in the second message middleware is acquired by an RCMC receiving module, the resource content data will be deleted from the second message middleware.

[0063] Optionally, after the RCMC receiving module obtains the resource content data, it can first perform basic verification and legality verification on the resource content data. After the verification passes, the resource content data is sent to the RCMC processing module. The method for verifying the resource content data is the same as the method for verifying resource content data by the writing module in the prior art, and will not be described in detail here.

[0064] Using this method, the RCMC receiving module can obtain resource content data from the second message middleware. Since the message middleware uses a sequential storage mechanism, the RCMC receiving module can obtain the resource content data in the order in which the resource content data is updated, so that the RCMC receiving module can subsequently send each resource content data to the RCMC processing module in the order in which the resource content data is updated.

[0065] In another embodiment of this disclosure, the resource content data received by the RCMC processing module in S201 above is: the resource content data corresponding to the hash value that has a preset mapping relationship with the RCMC processing module after the RCMC receiving module performs a consistent hash operation on the NID of the acquired resource content data to obtain a hash value.

[0066] Each time the RCMC receiving module receives a piece of resource content data, it performs a consistent hash operation on the NID of that resource content data to obtain the hash value of that NID. The RCMC receiving module pre-stores a preset mapping relationship between each hash value and an RCMC processing module. The RCMC receiving module can find the corresponding RCMC processing module for the calculated hash value from this preset mapping relationship, and then synchronize the resource content data to that RCMC processing module via the synchronization interface. Essentially, the RCMC receiving module can classify the received resource content data according to the hash value of the NID, ensuring that resource content data for the same resource is always processed by the same RCMC processing module.

[0067] For example, such as Figure 4 As shown, Figure 4 All three RCMC receiving modules can receive the resource content data of resource 1. These three RCMC receiving modules will send the resource content data belonging to resource 1 to the same RCMC processing module.

[0068] The consistent hashing operation used in this embodiment can be any consistent hashing algorithm, such as the division remainder method, which performs modulo operation on the NID and uses the remainder as the hash value of the NID.

[0069] When the number of RCMC processing modules in the system changes (e.g., the number of RCMC processing modules changes due to the addition or migration of RCMC instances), the mapping relationship between each hash value and the RCMC processing module can be redefined so that after the mapping relationship is changed, subsequent resource content data with the same NID will still be sent to the same RCMC processing module.

[0070] Using this method, the receiving function and the processing function are separated in this embodiment. The RCMC receiving module can send the resource content processing module with the same NID to the same RCMC processing module based on the mapping relationship between the hash value of NID and the RCMC processing module. This can avoid different RCMC processing modules processing different resource content data of the same resource, thereby avoiding the problem of timing disorder caused by the later updated resource content data being written to the cache layer first, so that the resource content data of the same resource is eventually written to the cache layer in sequence.

[0071] In one embodiment of this disclosure, the resource content data received by the RCMC processing module includes resource content data that needs to be written to the cache layer and resource content data that does not need to be written to the cache layer. Based on this, the above-mentioned S203 can be specifically implemented as follows:

[0072] After each piece of resource content data is written, if the resource content data needs to be written to the cache layer, the resource content data is added to the local queue corresponding to the first message middleware that the resource content data needs to be pushed. The multiple first message middlewares have a one-to-one correspondence with the multiple local queues in the RCMC processing module.

[0073] After writing resource content data to the database, RCMC can also determine whether the resource content data needs to be written to the cache layer. The determination method can be flexibly set according to actual needs. Resource content data that does not need to be distributed through the cache layer does not need to be written to the cache layer. For example, it can be set that resource content with a specified NID does not need to be written to the cache layer, or that resource content data of a specified type does not need to be written to the cache layer. This disclosure does not impose specific limitations on this aspect.

[0074] In related technologies, the writing module determines which first message middleware to push the resource content data to based on the actual content included in the resource content data. In this embodiment of the disclosure, the RCMC processing module has a local queue corresponding to each first message middleware. Therefore, the RCMC writing module can write resource content data that needs to be pushed to the same message middleware into the same first message middleware.

[0075] By using the above method, each time a piece of resource content data is written to the database, the resource data is added to the local queue corresponding to the first message middleware to which the resource content data needs to be pushed. This ensures that the resource content data that needs to be pushed to the same message middleware is written to the local queue in order, and then the subsequent resource content data can be pushed to the first message middleware in order, avoiding the time sequence disorder of the resource content data pushed to the first message middleware.

[0076] Based on the above embodiments, S204 can be specifically implemented as follows:

[0077] Each resource content data in a preset window of the local queue is obtained by a preset number of threads. The preset window includes the first preset number of resource content data in the local queue.

[0078] Each thread uses a mutex lock based on the NID of the acquired resource content data to push the acquired resource content data to the first message middleware corresponding to the local queue.

[0079] The size of the preset window can be flexibly configured, and the amount of resource content data included in the preset window of the local queue can be equal to the number of threads in the RCMC processing module that consume the resource content data in the local queue.

[0080] For example, such as Figure 5 As shown, assume that the RCMC processing module receives three resource content data in sequence, referred to as resource1", resource1' and resource2 respectively. These three resource content data need to be pushed to the same message middleware.

[0081] After the RCMC processing module writes resource 1 to the database, it can add resource 1 to... Figure 5 The local queue is shown. After writing resource 1' to the database, resource 1' is added to the local queue. After writing resource 2 to the database, resource 2 is added to the local queue.

[0082] Assuming the preset window is 3, the preset window of the local queue includes resource 1”, resource 1' and resource 2 in sequence. The RCMC module can push the resource content data in the preset window to the first message middleware through 3 threads.

[0083] For example, thread 1 acquires resource 1”, thread 2 acquires resource 1’, and thread 3 acquires resource 2. Because resources 1” and 1’ have the same NID, thread 1 uses the mutex lock of this NID when writing resource 1” to the first message middleware. At this time, thread 2 cannot process resource 1’. However, thread 3 can write resource 2 to the first message middleware. After thread 1’s resource 1” is written to the first message middleware, thread 2 can use the mutex lock of this NID to write resource 1 to the first message middleware. Thus, the resource content data in the first message middleware is resource 1”, resource 2, and resource 1’ in sequence.

[0084] Due to the existence of the mutex lock, the push order between resource 1" and resource 1' belonging to the same resource will not be disordered. It can be seen that this method can achieve global locking of resource content data at the resource granularity for database entry and caching. That is, the entire process uses a mutex lock with NID to sequentially write resource content data with the same NID to the database and push it to the first message middleware. This ensures that the order in which the cache layer retrieves resource content data with the same NID from the first message middleware is consistent with the storage order in the database, thereby guaranteeing the consistency of data versions stored in the cache layer and the database.

[0085] Corresponding to the above method embodiments, this disclosure also provides a data processing apparatus, which is applied to an RCMC processing module, such as... Figure 6 As shown, the device includes:

[0086] The receiving unit 601 is used to receive resource content data sent by the RCMC receiving module. The resource content data carries a message identifier NID, and the NID of the resource content data has a mapping relationship with the RCMC processing module.

[0087] The writing unit 602 is used to construct a mutex lock for the NID of the received resource content data, and write the resource content data carrying the same NID into the database sequentially based on the mutex lock;

[0088] Adding unit 603 is used to add the resource content data to the local queue after each resource content data is written;

[0089] The push unit 604 is used to push the resource content data in the local queue to the first message middleware corresponding to the local queue based on the mutex lock of the NID of each resource content data in the local queue, so that the caching service can obtain and cache the resource content data from the first message middleware.

[0090] In another embodiment of this disclosure, the pushing unit 604 is specifically used for:

[0091] Each resource content data in a preset window of the local queue is obtained by a preset number of threads. The preset window includes the first preset number of resource content data in the local queue.

[0092] Each thread uses a mutex lock based on the NID of the acquired resource content data to push the acquired resource content data to the first message middleware corresponding to the local queue.

[0093] In another embodiment of this disclosure, unit 603 is added, specifically for:

[0094] After each piece of resource content data is written, if the resource content data needs to be written to the cache layer, the resource content data is added to the local queue corresponding to the first message middleware that the resource content data needs to be pushed. The multiple first message middlewares have a one-to-one correspondence with the multiple local queues in the RCMC processing module.

[0095] In another embodiment of this disclosure, the resource content data received by the RCMC processing module is: the resource content data corresponding to the hash value that has a preset mapping relationship with the RCMC processing module after the RCMC receiving module performs a consistent hash operation on the NID of the acquired resource content data to obtain a hash value.

[0096] In another embodiment of this disclosure, the resource content data obtained by the RCMC receiving module comes from the second message middleware, which is used to cache the updated resource content data.

[0097] Corresponding to the above method embodiments, this disclosure also provides a data processing system, such as... Figure 7 As shown, the system includes: multiple RCMC receiving modules, multiple RCMC processing modules, a database, a first message middleware, and a caching service.

[0098] Figure 7 The example shows three RCMC receiving modules and three RCMC processing modules. In actual implementation, the number of modules included in the data processing system is not limited to this.

[0099] Each RCMC receiving module is used to send resource content data to the RCMC processing module that has a mapping relationship with the NID of the resource content data.

[0100] Each RCMC processing module is used to implement the method steps implemented by the RCMC processing module in the above method embodiments.

[0101] The caching service is used to retrieve and cache resource content data from the first message middleware.

[0102] The caching service can receive data query requests from business services. It prioritizes locally cached data. If local cached data is incomplete, such as the resource content data corresponding to a certain resource not being cached locally, the caching service can perform a database access to obtain the latest resource content data. However, if the locally cached data version is incorrect, the caching service will return the incorrect version of data to the business service without triggering the database access process. This would cause the business service to obtain the incorrect version of data, leading to business execution errors.

[0103] The system provided in this embodiment allows resource content data of the same resource to be sent to the same RCMC processing module. The RCMC processing module writes the resource content data of the same resource sequentially into the database and pushes it sequentially to the first message middleware based on a mutex lock. That is, for multiple resource content data of the same resource, they will be pushed strictly in the order of receipt, avoiding the cache service from obtaining the resource content data from the first message middleware in the wrong order, which would result in the cache of the wrong version of the resource content data. Under the premise of increasing system resource consumption as little as possible, the consistency of the data stored in the cache layer and the database in the system is improved.

[0104] In this embodiment of the disclosure, each RCMC receiving module is specifically used to perform a consistent hash operation on the NID of each acquired resource content data to obtain a hash value, and send the resource content data to the RCMC processing module that has a preset mapping relationship with the hash value.

[0105] Optionally, such as Figure 7 As shown, the system also includes a second message middleware.

[0106] The second message middleware is used to cache updated resource content data;

[0107] Multiple RCMC receiving modules are used to obtain resource content data from the second message middleware.

[0108] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0110] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0111] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0112] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0113] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0119] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0120] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, the method being applied to an RCMC processing module in a resource content management center, the RCMC processing module being located in a data processing system, the data processing system comprising multiple RCMC receiving modules, multiple RCMC processing modules, a database, a first message middleware, and a caching service; the method comprising: The system receives resource content data sent by the RCMC receiving module. The resource content data carries a message identifier (NID), and the NID of the resource content data has a mapping relationship with the RCMC processing module that receives the resource content data. Construct a mutex lock for the NID of the received resource content data, and write the resource content data with the same NID into the database sequentially based on the mutex lock; After each piece of resource content data is written, the resource content data is added to the local queue; Based on the mutex lock of the NID of each resource content data in the local queue, the resource content data in the local queue is pushed to the first message middleware corresponding to the local queue, so that the caching service can obtain and cache the resource content data from the first message middleware.

2. The method according to claim 1, wherein, Based on the mutex lock of the NID of each resource content data in the local queue, the resource content data in the local queue is pushed to the first message middleware corresponding to the local queue, including: Each resource content data in the local queue is obtained by a preset number of threads, wherein the preset window includes the first preset number of resource content data in the local queue; Each thread uses a mutex lock based on the NID of the acquired resource content data to push the acquired resource content data to the first message middleware corresponding to the local queue.

3. The method according to claim 1 or 2, wherein, After each piece of resource content data is written, the resource content data is added to the local queue, including: After each piece of resource content data is written, if the resource content data needs to be written to the cache layer, the resource content data is added to the local queue corresponding to the first message middleware that the resource content data needs to be pushed. The multiple first message middlewares have a one-to-one correspondence with the multiple local queues in the RCMC processing module.

4. The method according to claim 1, wherein, The resource content data received by the RCMC processing module is: the resource content data corresponding to the hash value that has a preset mapping relationship with the RCMC processing module after the RCMC receiving module performs a consistent hash operation on the NID of the acquired resource content data to obtain a hash value.

5. The method according to claim 4, wherein, The resource content data obtained by the RCMC receiving module comes from the second message middleware, which is used to cache the updated resource content data.

6. A data processing apparatus, wherein the apparatus is applied to an RCMC processing module of a resource content management center, the RCMC processing module being located in a data processing system, the data processing system comprising multiple RCMC receiving modules, multiple RCMC processing modules, a database, a first message middleware, and a caching service; the apparatus comprising: The receiving unit is used to receive resource content data sent by the RCMC receiving module. The resource content data carries a message identifier (NID), and the NID of the resource content data has a mapping relationship with the RCMC processing module that receives the resource content data. The write unit is used to construct a mutex lock for the NID of the received resource content data, and write the resource content data with the same NID into the database sequentially based on the mutex lock; The add unit is used to add the resource content data to the local queue after each write is completed; The push unit is used to push the resource content data in the local queue to the first message middleware corresponding to the local queue based on the mutex lock of the NID of each resource content data in the local queue, so that the caching service can obtain and cache the resource content data from the first message middleware.

7. The apparatus according to claim 6, wherein, The push unit is specifically used for: Each resource content data in the local queue is obtained by a preset number of threads, wherein the preset window includes the first preset number of resource content data in the local queue; Each thread uses a mutex lock based on the NID of the acquired resource content data to push the acquired resource content data to the first message middleware corresponding to the local queue.

8. The apparatus according to claim 6 or 7, wherein, The adding unit is specifically used for: After each piece of resource content data is written, if the resource content data needs to be written to the cache layer, the resource content data is added to the local queue corresponding to the first message middleware that the resource content data needs to be pushed. The multiple first message middlewares have a one-to-one correspondence with the multiple local queues in the RCMC processing module.

9. The apparatus according to claim 6, wherein, The resource content data received by the RCMC processing module is: the resource content data corresponding to the hash value that has a preset mapping relationship with the RCMC processing module after the RCMC receiving module performs a consistent hash operation on the NID of the acquired resource content data to obtain a hash value.

10. The apparatus according to claim 9, wherein, The resource content data obtained by the RCMC receiving module comes from the second message middleware, which is used to cache the updated resource content data.

11. A data processing system, comprising: Multiple RCMC receiving modules, multiple RCMC processing modules, database, first message middleware, and cache service; Each RCMC receiving module is used to send resource content data to the RCMC processing module that has a mapping relationship with the NID of the resource content data; Each RCMC processing module is configured to perform the method described in any one of claims 1-5; The caching service is used to obtain and cache resource content data from the first message middleware.

12. The system according to claim 11, wherein, Each RCMC receiving module is specifically used to perform a consistent hash operation on the NID of each acquired resource content data to obtain a hash value, and then send the resource content data to the RCMC processing module that has a preset mapping relationship with the hash value.

13. The system according to claim 12, further comprising a second message middleware; The second message middleware is used to cache updated resource content data; The plurality of RCMC receiving modules are used to obtain resource content data from the second message middleware.

14. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images