Data processing method, device, equipment and medium

Abstract

The present disclosure provides a data processing method, device, equipment and medium, which relates to the field of computer technology, and in particular to the field of data processing. The implementation scheme is: in response to receiving a write request for target data, storing the target data in a cache, and storing the target data in a message queue; reading the target data from the message queue; writing the target data read from the message queue into a database; and in response to receiving a query request, reading the target data from the cache and / or database.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of data processing, and specifically to a method, device, electronic device, computer-readable storage medium and computer program product for data processing. Background Art

[0002] The data processing system includes cache and database, and the data processing system may be distributed in different computer rooms, each computer room corresponds to a different cache and is connected to the same database. When the data flow is large, the performance of the database will be affected.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0005] According to one aspect of the present disclosure, a data processing method is provided, comprising: in response to receiving a write request for target data, storing the target data in a cache, and storing the target data in a message queue; reading the target data from the message queue; writing the target data read from the message queue to a database; and in response to receiving a query request, reading the target data from the cache and / or database.

[0006] According to another aspect of the present disclosure, a data processing device is provided, including: a cache configured to store data; a database configured to store data; a message queue configured to store data; a storage unit configured to store target data in the cache in response to receiving a write request for the target data; a producer configured to store the target data in the message queue in response to receiving a write request for the target data; a consumer configured to read the target data from the message queue; a write unit configured to write the target data read from the message queue to the database; and a query unit configured to read the target data from the cache and / or database in response to receiving a query request.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned data processing method.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above data processing method.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above data processing method when executed by a processor.

[0010] According to one or more embodiments of the present disclosure, the stability of a data processing system may be improved.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A flow chart showing a data processing method according to an exemplary embodiment of the present disclosure is shown;

[0014] Figure 2 A flow chart showing a data processing method according to an exemplary embodiment of the present disclosure is shown;

[0015] Figure 3 A structural block diagram of a data processing device according to an exemplary embodiment of the present disclosure is shown;

[0016] Figure 4 A schematic diagram showing a data processing process according to an exemplary embodiment of the present disclosure;

[0017] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0019] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0020] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0021] In the related art, the implementation method of data processing is to ensure the high availability of the database and synchronize the data to the database during the entire data processing process. The disadvantage of this implementation method is that when the data flow is large, frequent read and write operations will deteriorate the performance of the database. When the database is unavailable, it will block the entire data processing process and increase the maintenance cost of the database.

[0022] To solve the above problems, the present disclosure provides a data processing method, which synchronously writes the target data into the cache and introduces a message queue to decouple the database's write operation, so as to asynchronously write the target data into the database. The method introduces the cache to synchronously write the target data, and introduces the message queue to decouple the database's write operation, so as to reduce the pressure of peak data traffic on the database, improve the performance of data processing, and reduce the dependence of the data processing system on various components. When any one of the components of the cache, message queue and database is unavailable, data processing can still be performed normally, thereby improving the stability of the system.

[0023] The data processing method disclosed in the present invention can be applied to, but is not limited to, order processing systems. By adopting the method in the embodiments of the present invention, the entire order status flow and data are synchronously written into the cache, and the database is asynchronously written through the message queue, thereby reducing the pressure on the database and the dependence of the order process on various components. During the peak order traffic period, even if any component in the cache, message queue and database is unavailable, the order can be successfully placed, thereby improving the stability of the order processing system.

[0024] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 FIG. 4 is a flow chart showing a data processing method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the data processing method includes: step S101, in response to receiving a write request for target data, storing the target data in a cache, and storing the target data in a message queue; step S102, reading the target data from the message queue; step S103, writing the target data read from the message queue into a database; and step S104, in response to receiving a query request, reading the target data from the cache and / or database.

[0026] Therefore, by synchronously writing the target data into the cache and introducing the message queue to decouple the database's write operation, the target data can be asynchronously written to the database, thereby reducing the data processing system's dependence on various components. When the data flow is large, as long as the data is successfully written to at least one of the cache, message queue and database, the data processing can be guaranteed to be executed normally, thereby reducing the pressure on the database and improving the performance of data processing.

[0027] In one example, reading the target data from the cache and / or the database may be to compare the target data in the cache with the target data in the database, and use the target data in the latest state as the query result. When the target data cannot be read from the database, the target data in the cache is used as the query result. This enables querying the target data, reduces the pressure of large data traffic on the database, and thus improves the performance of data processing.

[0028] Exemplarily, the cache can be a redis cluster distributed cache, but is not limited thereto. For example, it can also be a non-relational database such as mongodb, CouchDB, etc. The message queue can be a rabbitMq message queue, and the database can be a MySQL relational database, but is not limited thereto. As long as the functions described above can be realized, the specific implementation forms of the cache, message queue, and database are not limited.

[0029] According to some embodiments, the message queue includes a delay queue and a target queue with a corresponding target consumer, the target data is stored in the target queue, and the target consumer is configured to automatically read the target data from the target queue. In this case, the message queue storing the target data may include: in response to the target data read from the target queue failing to be written to the database, sending the target data to the delay queue; and in response to satisfying a first preset condition, sending the target data in the delay queue to the target queue again.

[0030] Therefore, when the target data automatically read from the target queue fails to be written to the database, the target data is first written to the delay queue to wait for retrying to write to the database, thereby avoiding target data loss and protecting data integrity, which can fully meet the needs of actual application scenarios.

[0031] In some embodiments, the target data in the delay queue may be sent to the target queue again at intervals, waiting to be automatically consumed by the target consumer, so as to retry writing to the library.

[0032] Accordingly, according to some embodiments, the message queue stores the target data including: in response to each failure of writing the target data to the database, recording the corresponding start time when the target data is sent to the delay queue, wherein the first preset condition includes: the time interval between the current time and the start time reaches a preset duration.

[0033] The first preset condition is equivalent to the waiting time of the target data in the delay queue reaching the preset threshold. The first preset condition can be flexibly adjusted according to the needs of the actual application scenario, thereby improving the performance of data processing. For example, a threshold for the number of times data is retried to write to the database can be set to avoid wireless cyclic retries to write the target data to the database, which increases the pressure on the database.

[0034] Based on this, according to some embodiments, the message queue includes a dead letter queue, the data in the dead letter queue will not be automatically consumed, and wherein the message queue stores the target data including: in response to the cumulative number of times the target data is sent from the target queue to the delay queue reaching a threshold, sending the target data to the dead letter queue. Thus, when the target data fails to be written to the database multiple times, it can be temporarily stored in the dead letter queue, which avoids the loss of the target data and reduces the pressure on the database during the peak period of data traffic, thereby improving the performance of data processing.

[0035] Further, according to some embodiments, reading the target data from the message queue in step S102 includes: in response to satisfying a second preset condition, triggering a dead letter consumer to read the target data from the dead letter queue; and writing the target data read from the message queue to the database in step S103 includes: writing the target data read from the dead letter queue to the database. Thus, data loss can be effectively avoided and data integrity can be fully protected.

[0036] According to some embodiments, the second preset condition includes that the database is in a writable data state. The writable data state may be an unavailable fault state and a recovery state of the corresponding database in an extreme scenario, thereby effectively avoiding data loss and fully protecting data integrity.

[0037] It is understandable that the second preset condition is not limited to this. For example, the second preset condition may include that the target data flow is less than a certain threshold, thereby balancing the data processing operations during the peak and trough periods of data flow, making full use of the hardware performance of database reading and writing, and improving the performance of data processing.

[0038] Exemplarily, in response to satisfying the second preset condition, the dead letter consumer may be manually triggered to read the target data from the dead letter queue.

[0039] According to some embodiments, the data processing method further comprises: in response to a failure to store the target data in the message queue, directly writing the target data into the database. Thus, the target data can be synchronously written into the database when the message queue is unavailable, reducing the dependence of the data processing system on the message queue and improving the stability of the system.

[0040] According to some embodiments, in response to receiving a query request, reading the target data from the cache and / or database includes: in response to a failure to store the target data in the cache, reading the target data from the database. Thus, when the cache is unavailable, the database can be directly queried, which reduces the dependence of the data processing system on the cache and improves the stability of the system.

[0041] According to some embodiments, in response to receiving a query request, reading the target data from the cache and / or database includes: in response to not reading the target data from the database, reading the target data from the cache. Thus, the cache can be used to implement the query when the database is unavailable, reducing the dependence of the data processing system on the database and improving the stability of the system. At the same time, multiple attempts to query the database can be avoided, reducing the pressure on the database.

[0042] According to some embodiments, the cache includes multiple sub-caches, and storing the target data in the cache in step S101 includes: storing the target data in each sub-caches. Thus, when any sub-caches are unavailable, data can be queried from other sub-caches, especially when the database is unavailable, which can avoid affecting the performance of data query, further reduce the dependence on the sub-caches, and thus improve the performance and stability of the system.

[0043] According to some embodiments, the target data includes a data state, and in response to receiving a query request in step S104, reading the target data from the cache and / or database includes: in response to reading the target data corresponding to the query request from both the cache and the database, based on the data state of the target data, determining the target data with the latest data state among the target data read from the cache and the database as the query result. Thus, the target data with the latest data state can be used as the query result, ensuring the accuracy of the query result.

[0044] According to some embodiments, when the target data is order data, the data processing system may be an order processing system, and the processing of the order data is realized accordingly. The pressure on the order database is reduced during the peak consumption period, thereby fully meeting the needs of the actual application scenario. Exemplarily, the data status of the order data may include order status, such as pending payment status, paid status, canceled order status, etc.

[0045] According to some embodiments, when the target data is order data, the target data includes a target order status, and the method further includes: before writing the target data into the database, obtaining the latest order status related to the target data in the cache and / or database, wherein writing the target data into the database is performed in response to the latest order status being a pre-state of the target order status. Thus, the correctness of the target data written into the database can be ensured by checking the preset order status flow conditions, avoiding writing data with incorrect order status into the database, ensuring the correctness and integrity of the data, and thus being able to fully meet the needs of actual application scenarios.

[0046] According to some embodiments, when the target data is order data, the target data includes a target order status, and the message queue includes a delay queue, the data processing method further includes: in response to the latest order status not being a pre-state of the target order status, sending the target data to the delay queue. Thus, by temporarily storing data with incorrect order status in the delay queue to wait for a retry to write to the library, the correctness and integrity of the data can be guaranteed, thereby fully meeting the needs of actual application scenarios.

[0047] In one example, the target order status can be used to represent the payment status corresponding to the order, which can be a pending payment status, a paid status, or a canceled order status, etc. The writing of the target data into the database is performed in response to the latest order status being a predecessor status of the target order status, and the write operation can be performed based on a preset order status flow condition. For example, when the target order status is a paid status, the write operation to the target data can only be performed when the latest order status related to the target data in the cache and / or database is a pending payment status, and when the latest order status is an order canceled status, the corresponding write operation cannot be performed. Thus, the correctness of the target data written into the database can be guaranteed by verifying the preset order status flow conditions, fully meeting the needs of actual application scenarios.

[0048] According to some embodiments, in the case where the target data is order data, the data status includes the order status, and in response to receiving the query request in step S104, reading the target data from the cache and / or database includes: in response to reading the order data corresponding to the query request from both the cache and the database, based on the order status of the order data, determining the order data with the latest order status from the order data read from the cache and the database as the query result. Thus, the order data with the latest order status can be used as the query result, ensuring the accuracy of the order query result.

[0049] Figure 2 A flow chart of a data processing method according to an exemplary embodiment of the present disclosure is shown.

[0050] like Figure 2 As shown, the data processing method includes:

[0051] Step S201: In response to receiving a write request for target data, storing the target data in a cache, and storing the target data in a target queue in a message queue, wherein the message queue further includes a delay queue and a dead letter queue;

[0052] Step S202: The target consumer automatically reads the target data from the target queue;

[0053] Writing the target data read from the target queue into a database, and determining whether the target data is successfully stored in the database;

[0054] In response to the target data being successfully stored in the database, the process ends.

[0055] In response to the target data being stored in the database unsuccessfully, determining whether the number of times the target data is sent from the target queue to the delay queue reaches a threshold;

[0056] In response to the number of times the target data is sent from the target queue to the delay queue not reaching the threshold, executing step S203, sending the target data to the delay queue;

[0057] In response to satisfying the first preset condition (for example, the duration of the target data being sent to the delay queue reaches a preset value), executing step S204, sending the target data in the delay queue to the target queue again to wait for being automatically read by the target consumer again;

[0058] In response to the number of times the target data is sent from the target queue to the delay queue reaching a threshold, executing step S205 to send the target data to the dead letter queue;

[0059] In response to satisfying the second preset condition (for example, the database is restored to be writable), step S206 is executed to manually trigger the dead letter consumer to read the target data from the dead letter queue, and then step S207 is executed to write the target data read from the dead letter queue into the database.

[0060] According to another aspect of the present disclosure, a data processing device is also provided. Figure 3 A structural block diagram of a data processing device according to an exemplary embodiment of the present disclosure is shown. Figure 3 As shown, the data processing device 300 includes: a cache 301, configured to store data; a database 302, configured to store data; a message queue 303, configured to store data; a storage unit 304, configured to store the target data to the cache 301 in response to receiving a write request for the target data; a producer 305, configured to store the target data to the message queue 303 in response to receiving a write request for the target data; a consumer 306, configured to read the target data from the message queue 303; a write unit 307, configured to write the target data read from the message queue 303 to the database 302; and a query unit 308, configured to read the target data from the cache 301 and / or the database 302 in response to receiving a query request. Therefore, by synchronously writing the target data into the cache and introducing the message queue to decouple the database's write operation, the target data can be asynchronously written to the database, which can reduce the data processing system's dependence on various components. When the data flow is large, as long as the data is successfully written to at least one of the cache, message queue and database, the data processing can be guaranteed to be executed normally, thereby reducing the pressure on the database and improving the performance of data processing.

[0061] According to some embodiments, the consumer 306 includes a target consumer, the message queue 303 includes a delay queue and a target queue with a corresponding target consumer, the producer 305 is configured to store the target data in the target queue, the target consumer is configured to automatically read the target data from the target queue, the write unit is configured to write the target data read from the target queue into a database, and the message queue 303 also includes: a first sending unit, configured to send the target data to the delay queue in response to the failure of the target data to be written into the database; and a second sending unit, configured to send the target data in the delay queue to the target queue again in response to satisfying a first preset condition. The specific content of the first preset condition is consistent with that described above and will not be repeated here. As a result, when the target data fails to be written into the database, it is possible to retry to write the target data into the database, avoid the loss of the target data, and protect the data integrity, so as to fully meet the needs of actual application scenarios.

[0062] According to some embodiments, the message queue 303 includes a dead letter queue, and the data in the dead letter queue will not be automatically consumed, and the message queue 303 also includes: a third sending unit, configured to send the target data to the dead letter queue in response to the cumulative number of times the target data is sent from the target queue to the delay queue reaching a threshold. In this way, the target data can be temporarily stored when it fails to be written to the database multiple times, which avoids the loss of the target data and reduces the pressure on the database during the peak period of data traffic, thereby improving the performance of data processing.

[0063] According to some embodiments, the consumer 306 further includes: a dead letter consumer configured to be triggered to read the target data from the dead letter queue, wherein the writing unit 307 is further configured to write the target data read from the dead letter queue into a database. Thus, data loss can be effectively avoided and data integrity can be fully protected.

[0064] Figure 4 FIG. 1 is a schematic diagram showing a data processing process according to an exemplary embodiment of the present disclosure, wherein arrows show the direction of signal flow. Figure 4As shown, the operation request and / or the flow direction of the target data between the modules, the data processing process may include the following steps: Step S11, in response to receiving a write request for the target data, the target data is synchronously stored in the cache 301, and the producer 305 stores the target data in the target queue in the message queue 303, and the message queue 303 includes a target queue, a delay queue and a dead letter queue; Step S12, the target consumer automatically reads the target data from the target queue, and the target data is written to the database 302; Step S13, the target data is written to the database. Step S14, in response to the target data being stored in the database unsuccessfully, the target data is sent to the delay queue; Step S15, in response to satisfying a first preset condition, the first preset condition may be, for example, that the time interval between the current time and the start time when the target data is sent to the delay queue reaches a preset duration, the target data in the delay queue is sent to the target queue again, waiting to be read again by the target consumer to be written into the database 302; Step S16, if the target data is not successfully written into the database for multiple times, in response to the cumulative number of times the target data is sent from the target queue to the delay queue reaching a threshold, the target data is sent to the dead letter queue; Step S17, in response to satisfying a second preset condition, the target data is read from the dead letter queue by manually triggering the dead letter consumer, and the target data is written into the database 302; and Step S18, in response to the target data being stored in the message queue unsuccessfully, the target data is directly written into the database. Thus, as long as the data is successfully written into at least any one of the cache, the message queue and the database, the normal execution of data processing can be guaranteed, thereby improving the performance and stability of the data processing system.

[0065] The data processing process also includes: step S19, in response to receiving the query request, reading the target data from the cache 301 and the database 302, comparing the target data in the cache 301 and the database 302, and taking the target data in the latest data state as the query result.

[0066] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned data processing method.

[0067] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned data processing method.

[0068] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the above data processing method when executed by a processor.

[0069] See also Figure 5 , a block diagram of an electronic device 500 that can be used as the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be a computer device of different types, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0070] Figure 5 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Figure 5 As shown, the electronic device 500 may include at least one processor 501 , a working memory 502 , an I / O device 504 , a display device 505 , a storage device 506 , and a communication interface 507 that can communicate with each other via a system bus 503 .

[0071] The processor 501 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 501 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. The processor 501 may be configured to obtain and execute computer-readable instructions stored in a working memory 502, a storage device 506, or other computer-readable media, such as program codes of an operating system 502a, program codes of an application 502b, and the like.

[0072] Working memory 502 and storage device 506 are examples of computer-readable storage media for storing instructions, which are executed by processor 501 to implement the various functions described above. Working memory 502 may include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, storage device 506 may include a hard disk drive, a solid-state drive, a removable medium, including external and removable drives, memory cards, flash memory, a floppy disk, an optical disk (e.g., CD, DVD), a storage array, a network attached storage, a storage area network, etc. Working memory 502 and storage device 506 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 501 as a specific machine configured to implement the operations and functions described in the examples herein.

[0073] The I / O device 504 may include an input device and / or an output device. The input device may be any type of device capable of inputting information to the electronic device 500, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output device may be any type of device capable of presenting information, and may include but is not limited to a video / audio output terminal, a vibrator, and / or a printer.

[0074] The communication interface 507 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0075] The application 502b in the working register 502 can be loaded to execute the various methods and processes described above, such as Figure 1 In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the storage device 506 and / or the communication interface 507. When the computer program is loaded and executed by the processor 501, one or more steps of the data processing method described above may be performed.

[0076] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0078] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0080] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0081] A computing system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0082] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

[0083] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing method, comprising: In response to receiving a write request for target data, storing the target data in a cache and storing the target data in a message queue, wherein the target data is order data including a data status, and the data status includes an order status; Reading the target data from the message queue; Writing the target data read from the message queue into a database; and In response to receiving the query request, reading the target data from the cache and / or the database, Wherein, the method further comprises: Before writing the target data into the database, obtaining the latest order status related to the target data in the cache and / or the database, And wherein, writing the target data into the database is performed in response to the latest order status being a predecessor state of the target order state of the target data.

2. The method according to claim 1, wherein: The message queue includes a delay queue and a target queue having a corresponding target consumer, the target data is stored in the target queue, and the target consumer is configured to automatically read the target data from the target queue, And wherein, the message queue storing the target data comprises: In response to failure in writing the target data read from the target queue into the database, sending the target data to the delay queue; and In response to satisfying a first preset condition, the target data in the delay queue is sent to the target queue again.

3. The method according to claim 2, wherein: The message queue includes a dead letter queue, and the data in the dead letter queue will not be automatically consumed. And wherein, the message queue storing the target data comprises: In response to the cumulative number of times that the target data is sent from the target queue to the delay queue reaching a threshold, the target data is sent to the dead letter queue.

4. The method according to claim 3, wherein: Reading the target data from the message queue includes: In response to satisfying a second preset condition, triggering a dead letter consumer to read the target data from the dead letter queue, Wherein, writing the target data read from the message queue into a database comprises: The target data read from the dead letter queue is written into a database.

5. The method according to claim 4, wherein: The second preset condition includes that the database is in a data-writable state.

6. The method according to claim 2, wherein: The message queue storing the target data includes: In response to each failure of writing the target data into the database, recording the corresponding start time when the target data is sent to the delay queue, The first preset condition includes: the time interval between the current time and the start time reaches a preset duration.

7. The method according to any one of claims 1 to 6, further comprising: In response to failure to store the target data in the message queue, directly writing the target data into the database.

8. The method according to any one of claims 1 to 6, wherein: In response to receiving the query request, reading the target data from the cache and / or the database includes: In response to failure to store the target data in the cache, reading the target data from the database.

9. The method according to any one of claims 1 to 6, wherein: In response to receiving the query request, reading the target data from the cache and / or the database includes: In response to not reading the target data from the database, reading the target data from the cache.

10. The method according to any one of claims 1 to 6, wherein: The cache includes a plurality of sub-caches, and wherein storing the target data in the cache includes: The target data is stored in each sub-buffer.

11. The method according to any one of claims 1 to 6, wherein: In response to receiving the query request, reading the target data from the cache and / or the database includes: In response to target data corresponding to the query request being read from both the cache and the database, based on the data status of the target data, the target data with the latest data status among the target data read from the cache and the database is determined as the query result.

12. The method according to any one of claims 2 to 6, further comprising: In response to the latest order status not being a preceding status of the target order status, sending the target data to the delay queue.

13. The method according to claim 1, wherein: In response to receiving the query request, reading the target data from the cache and / or the database includes: In response to order data corresponding to the query request being read from both the cache and the database, based on the order status of the order data, order data with the latest order status among the order data read from the cache and the database is determined as the query result.

14. A data processing device, comprising: Cache, which is configured to store data; A database, configured to store data; Message queues, which are configured to store data; A storage unit configured to store the target data into the cache in response to receiving a write request for the target data; A producer is configured to store the target data in a message queue in response to receiving a write request for the target data, wherein the target data is order data including a data state, and the data state includes an order state; A consumer, configured to read the target data from the message queue; a writing unit configured to write the target data read from the message queue into a database; and a query unit, configured to read the target data from the cache and / or the database in response to receiving a query request, Wherein, the writing unit is further configured as: Before writing the target data into the database, obtaining the latest order status related to the target data in the cache and / or the database, And wherein, writing the target data into the database is performed in response to the latest order status being a predecessor state of the target order state of the target data.

15. The device according to claim 14, wherein: The consumer includes a target consumer, the message queue includes a delay queue and a target queue with a corresponding target consumer, the producer is configured to store the target data in the target queue, the target consumer is configured to automatically read the target data from the target queue, and the writing unit is configured to write the target data read from the target queue into a database, And wherein the message queue also includes: A first sending unit, configured to send the target data to the delay queue in response to a failure in writing the target data into the database; and The second sending unit is configured to send the target data in the delay queue to the target queue again in response to satisfying a first preset condition.

16. The device according to claim 15, wherein: The message queue includes a dead letter queue, data in the dead letter queue will not be automatically consumed, and the message queue further includes: The third sending unit is configured to send the target data to the dead letter queue in response to the accumulated number of times that the target data is sent from the target queue to the delay queue reaching a threshold.

17. The device according to claim 16, wherein: The consumers also include: The dead letter consumer is configured to be triggered to read the target data from the dead letter queue in response to satisfying a second preset condition, The writing unit is further configured to write the target data read from the dead letter queue into a database.

18. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

20. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images