A data processing method, apparatus, computer equipment, and storage medium

Abstract

This invention provides a data processing method, apparatus, computer device, and storage medium. The method includes: responding to a client's request to retrieve a target questionnaire, obtaining the version number of the target questionnaire; retrieving questionnaire data of the target questionnaire from a multi-level storage space based on the version number of the target questionnaire, wherein the multi-level storage space includes one or more of local memory space, distributed cache space, and persistent storage space; and sending the questionnaire data of the target questionnaire to the client. Multi-level caching ensures fast response in high-concurrency questionnaire reading scenarios, and the version number ensures data timeliness and consistency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] For example, in scenarios where users participate in live quiz shows, current systems like Wenjuanxing and Wenjuanwang host data on external devices, which is uncontrollable, insecure, and doesn't support high-concurrency data access. Clients often have customized needs, such as quiz activities during live games, where celebrities and influencers share questionnaire links in their live streams, offering rewards to the top few respondents. In these scenarios, all participating users answer questions simultaneously, requiring high concurrency for viewing, filling out, and submitting questionnaires—features that existing questionnaire systems don't support. Therefore, providing a secure and reliable questionnaire management method that supports high-concurrency scenarios has become an urgent problem to solve. Summary of the Invention

[0003] This invention provides a data processing method, apparatus, computer equipment, and storage medium that can ensure rapid response in high-concurrency questionnaire reading scenarios and ensure the timeliness and consistency of data by adding version numbers.

[0004] In a first aspect, embodiments of the present invention provide a data processing method, the method comprising:

[0005] In response to a client's request to retrieve a target questionnaire, obtain the version number of the target questionnaire.

[0006] The questionnaire data of the target questionnaire is obtained from a multi-level storage space according to the version number of the target questionnaire. The multi-level storage space includes one or more of local memory space, distributed cache space and persistent storage space.

[0007] The questionnaire data of the target questionnaire is sent to the client.

[0008] In a second aspect, embodiments of the present invention provide a data processing apparatus, the apparatus comprising:

[0009] The acquisition module is used to obtain the version number of the target questionnaire in response to a client's questionnaire acquisition request for the target questionnaire.

[0010] The acquisition module is further configured to acquire the questionnaire data of the target questionnaire from a multi-level storage space according to the version number of the target questionnaire, wherein the multi-level storage space includes one or more of local memory space, distributed cache space and persistent storage space.

[0011] The sending module is used to send the questionnaire data of the target questionnaire to the client.

[0012] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a network interface and a storage device, the processor, the network interface and the storage device being interconnected, wherein the network interface is controlled by the processor for sending and receiving data, the storage device is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions for executing the data processing method described in the first aspect.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which are executed by a processor to perform the data processing method described in the first aspect.

[0014] Fifthly, this invention discloses a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described in the first aspect.

[0015] In this embodiment of the invention, the working node can respond to the client's request to retrieve the target questionnaire, obtain the version number of the target questionnaire, and retrieve the questionnaire data of the target questionnaire from a multi-level storage space based on the version number of the target questionnaire. The multi-level storage space includes one or more of local memory space, distributed cache space, and persistent storage space. After reading the questionnaire data from any level of storage space in the multi-level storage space, the node can send the questionnaire data of the target questionnaire to the client. Thus, multi-level caching ensures fast response in high-concurrency questionnaire reading scenarios, and the addition of version numbers ensures data timeliness and data consistency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1a This is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present invention;

[0019] Figure 1b This is a schematic diagram of a questionnaire management architecture provided by an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0021] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention;

[0022] Figure 4a This is a schematic diagram of a questionnaire publishing process provided by an embodiment of the present invention;

[0023] Figure 4b This is a flowchart illustrating a process for viewing a questionnaire, provided by an embodiment of the present invention.

[0024] Figure 4c This is a schematic diagram of a questionnaire submission process provided by an embodiment of the present invention;

[0025] Figure 4d This is a schematic diagram of a process for persistent storage of questionnaire data provided in an embodiment of the present invention;

[0026] Figure 4e This is a schematic diagram of another process for persistent storage of questionnaire data provided in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] See Figure 1a This is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present invention. The data processing system includes a client 10, a worker node 20, a distributed cache node 30, a persistent storage node 40, and a message queue node 50, wherein:

[0031] Worker node 20 can be a distributed computer device (such as a server) used to provide services such as questionnaire publishing, viewing, and answer collection. For example, worker node 20 can communicate with client 10, receive questionnaire data submitted by client 10, and store the questionnaire data in multi-level storage space, such as local memory space, distributed cache space, persistent storage space, etc. When client 10 submits a questionnaire viewing request, worker node 20 can efficiently read data from multi-level storage space, thereby supporting high-concurrency questionnaire reading scenarios. Work node 20 can also receive questionnaire data submitted by client 10. Before writing the questionnaire data to persistent storage, in order to ensure the effective storage of a large number of questionnaire data submitted by users in high-concurrency scenarios and to prevent data loss, the questionnaire data can be written to a distributed cache space and a message middleware queue. The distributed cache space can be provided by database software, such as a distributed cache space, and the message middleware queue can be provided by a distributed log system. By using the distributed message middleware distributed log system, the submitted questionnaire data can be processed asynchronously based on reliability. For example, the user's questionnaire data can be asynchronously retrieved from the message middleware queue of the distributed log system and then written to persistent storage.

[0032] Client 10 can be used by users and questionnaire publishers. For example, users can use Client 10 to conduct questionnaires, submit their answers, and view the results. Questionnaire publishers can use Client 10 to publish questionnaires, such as publishing new questionnaires or updating previously published questionnaires.

[0033] Distributed cache node 30 can provide a distributed cache space, which is used to cache questionnaire data, answer sheet data, questionnaire version number, etc.

[0034] Persistent storage node 40 can provide persistent storage space, such as a database DB, for long-term storage of questionnaire data, answer sheet data, questionnaire version number, etc.

[0035] Message queue node 50 can provide a message middleware queue based on a distributed log system. This message middleware queue can be used to cache user-submitted questionnaire data, wait for the data consumer (i.e., worker node 20) to retrieve the questionnaire data from the message middleware queue, and asynchronously write it to the persistent storage space, ensuring the security and effective storage of the data.

[0036] The worker node 20, distributed cache node 30, persistent storage node 40, and message queue node 50 can be distributed cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The client 10 can specifically be a smartphone, tablet, laptop, desktop computer, in-vehicle intelligent terminal, etc., and this embodiment of the invention is not limited thereto.

[0037] In some feasible implementations, the process for obtaining questionnaire data can be as follows: The user sends a questionnaire retrieval request for the target questionnaire to the worker node 20 through the client 10. The worker node 20 responds to the questionnaire retrieval request, obtains the version number of the target questionnaire, and retrieves the questionnaire data of the target questionnaire from a multi-level storage space based on the version number of the target questionnaire. The multi-level storage space includes one or more of local memory space, distributed cache space, and persistent storage space. After reading the questionnaire data from any level of storage space in the multi-level storage space, the questionnaire data of the target questionnaire can be sent to the client. Thus, the multi-level caching ensures a fast response in high-concurrency questionnaire reading scenarios, and the version number ensures data timeliness and data consistency.

[0038] See Figure 1b This is based on the embodiments of the present invention. Figure 1a The data processing system shown provides an architecture diagram for questionnaire management, in which the questionnaire management architecture can be divided into a user layer 60, a front-end page layer 70, a collection service layer 80, and a storage layer 90.

[0039] Specifically, the user layer 60 can include questionnaire administrators and respondents; the front-end page layer 70 provides various operation pages required for interaction with administrators and respondents, such as questionnaire editing pages, questionnaire completion pages, and answer viewing pages. Administrators publish new questionnaires or update previously published questionnaires through the questionnaire editing page; respondents submit their answers through the questionnaire completion page and can view the relevant answers through the answer viewing page. The collection service layer 80 provides various Common Gateway Interfaces (CGIs) required in the questionnaire management process, including questionnaire publishing CGI, questionnaire retrieval CGI, answer submission CGI, and answer viewing CGI, as well as questionnaire data consumers and answer data consumers. Questionnaire data consumers and answer data consumers refer to the worker nodes being able to write questionnaire data, answer data, and other data into various storage spaces in the storage layer; the storage layer 90 provides multi-level storage spaces, including distributed cache space, message middleware queues, persistent storage space (such as databases (DB)), etc. In addition, the memory space of the worker nodes can also provide data caching services.

[0040] The implementation details of the technical solutions in the embodiments of the present invention are described in detail below:

[0041] See Figure 2 This is based on the embodiments of the present invention. Figure 1a The illustrated data processing system provides a flowchart of a data processing method. The data processing method of this embodiment is mainly described from the perspective of the working node, and includes the following steps:

[0042] 201. In response to the client's request to retrieve the target questionnaire, obtain the version number of the target questionnaire.

[0043] Specifically, when a worker node receives a questionnaire retrieval request submitted by a client, it can obtain the identification information of the target questionnaire carried in the questionnaire retrieval request and send a version retrieval request to the first storage node. The version retrieval request carries the identification information of the target questionnaire. The first storage node corresponds to the distributed cache node 30 mentioned above, and can be... Figure 1a Any one of the distributed cache nodes in the process is used to provide distributed cache space; the first storage node responds to the version retrieval request and retrieves the version number of the target questionnaire from the distributed cache space according to the identification information of the target questionnaire, such as the latest version number, and sends the version number of the target file to the worker node.

[0044] 202. Obtain the questionnaire data of the target questionnaire from a multi-level storage space according to the version number of the target questionnaire. The multi-level storage space includes one or more of local memory space, distributed cache space and persistent storage space.

[0045] This application can provide multi-level storage space to store questionnaire data. Specifically, the multi-level storage space can include local memory space, distributed cache space, and persistent storage space. The local memory space is provided by worker nodes, the distributed cache space is provided by distributed cache nodes, and the persistent storage space is provided by persistent storage nodes.

[0046] Specifically, after obtaining the version number of the target questionnaire, the worker node can retrieve the questionnaire data from the local memory space, distributed cache space, or persistent storage space according to the set query order. The set query order can be local memory space first, then distributed cache space, and then persistent storage space. That is, the questionnaire data is retrieved from the storage space with better read performance first. Using multi-level storage space to provide questionnaire data can ensure efficient reading in high-concurrency scenarios.

[0047] 203. Send the questionnaire data of the target questionnaire to the client.

[0048] Specifically, after reading the questionnaire data from any level of storage space in the multi-level storage space according to the set query order, the worker node can send the questionnaire data of the target questionnaire to the client, thereby ensuring fast response in high-concurrency questionnaire reading scenarios through multi-level caching.

[0049] In this embodiment of the invention, the working node can respond to the client's request to retrieve the target questionnaire, obtain the version number of the target questionnaire, and retrieve the questionnaire data of the target questionnaire from a multi-level storage space based on the version number of the target questionnaire. The multi-level storage space includes one or more of local memory space, distributed cache space, and persistent storage space. After reading the questionnaire data from any level of storage space in the multi-level storage space, the node can send the questionnaire data of the target questionnaire to the client. Thus, multi-level caching ensures fast response in high-concurrency questionnaire reading scenarios, and the addition of version numbers ensures data timeliness and data consistency.

[0050] See Figure 3 This is based on the embodiments of the present invention. Figure 1a The diagram shows another data processing method provided by the data processing system. The data processing method of this embodiment is mainly described from the perspective of the working node, and includes the following steps:

[0051] 301. In response to the client's request to retrieve the target questionnaire, obtain the version number of the target questionnaire.

[0052] The specific implementation of step 301 can be found in the relevant description of step 201 in the aforementioned embodiments, and will not be repeated here.

[0053] 302. Query whether the local memory space contains the questionnaire data of the target questionnaire according to the version number of the target questionnaire. If the local memory space contains the questionnaire data of the target questionnaire, then read the questionnaire data of the target questionnaire from the local memory space according to the version number of the target questionnaire.

[0054] Specifically, considering that local memory space has the best data read performance, distributed cache space has the second best data read performance, and persistent storage space has the worst data read performance, after the worker node obtains the version number of the target file, it can first check whether the target questionnaire data is cached in the local memory space. If it is, the target questionnaire data can be read directly from the local memory space.

[0055] In some feasible implementations, the worker node can first check whether the version number of the questionnaire data cached in the local memory space matches the version number of the target questionnaire. If they match, it means that the cached data is the latest questionnaire data and can be read directly. If they do not match, it means that the cached data is not the latest questionnaire data. In this case, it is necessary to try to read the questionnaire data of the target file from the distributed cache space of the distributed cache node.

[0056] 303. If the local memory space does not include the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the distributed cache space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the local memory space.

[0057] Specifically, if the local memory space does not contain the target questionnaire data, the worker node can query the distributed cache space of the first storage node (i.e., the aforementioned distributed cache node) based on the version number of the target questionnaire to see if it contains the target questionnaire data. If the distributed cache space contains the target questionnaire data, the worker node receives the target questionnaire data sent by the first storage node. Furthermore, the worker node can write the target questionnaire data into its local memory space. When other users need to read the target questionnaire data, the worker node can quickly read the questionnaire data from its local memory space and provide it to the user.

[0058] 304. If the distributed cache space does not include the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the persistent storage space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the distributed cache space and the local memory space.

[0059] Specifically, if the distributed cache space also does not include the target questionnaire data, considering that the distributed cache space can be written to by multiple worker nodes, to ensure data consistency, the current worker node can set a distributed lock for the target questionnaire, preventing other worker nodes from temporarily operating on the target questionnaire data, such as reading the questionnaire data or writing the questionnaire data to the distributed cache space. The current worker node can query the persistent storage space of the second storage node based on the version number of the target questionnaire to see if it includes the target questionnaire data. The second storage node corresponds to the persistent storage node mentioned above, and can be any persistent storage node. If the persistent storage space includes the target questionnaire data, the worker node receives the target questionnaire data sent by the second storage node. After obtaining the target questionnaire data, it releases the distributed lock, allowing other worker nodes to regain the permission to operate on the target questionnaire data. The worker node then writes the target questionnaire data to the distributed cache space and the local memory space. This allows other users' questionnaire retrieval requests to preferentially read questionnaire data from the local memory space or the distributed cache space, improving the questionnaire data reading performance and meeting the data reading requirements in high-concurrency scenarios.

[0060] In some feasible implementations, if the persistent storage space does not include the questionnaire data of the target questionnaire, the distributed lock is released, and the current read request can be regarded as a malicious attack. At this time, empty data corresponding to the version number of the target questionnaire can be written to the distributed cache space. When the attacker tries to read the data again, the corresponding empty data can be read from the distributed cache space and returned to the attacker, thus ensuring data security.

[0061] In some feasible implementations, before the current working node sets a distributed lock on the target questionnaire, considering that other working nodes may have already read the questionnaire data of the target questionnaire from the persistent storage space and written it to the distributed cache space, the current working node can query the distributed cache space again to see if it contains the questionnaire data of the target questionnaire after setting the distributed lock on the target questionnaire. For example, it can query the distributed cache space of the first storage node to see if it contains the questionnaire data of the target questionnaire based on the version number of the target questionnaire. If the distributed cache space contains the questionnaire data of the target questionnaire at this time, it can receive the questionnaire data of the target questionnaire sent by the first storage node, then release the distributed lock, and write the questionnaire data of the target questionnaire to the local memory space. That is, it is not necessary to obtain the questionnaire data of the target questionnaire from the persistent storage space again, thus improving the data reading speed. If the distributed cache space still does not contain the questionnaire data of the target questionnaire at this time, then it is necessary to query the persistent storage space of the second storage node based on the version number of the target questionnaire to obtain the questionnaire data of the target questionnaire.

[0062] 305. Receive the answer data of the target questionnaire sent by the client through the answer submission interface.

[0063] Specifically, after completing the questionnaire, users submit their answers through the client. For example, the client can submit the answers to the target questionnaire to the worker node through the answer submission interface (CGI).

[0064] 306. Write the answer data of the target questionnaire to the distributed cache space of the first storage node and the first message middleware queue of the third storage node in the order of writing.

[0065] Specifically, since user filling in and submitting answers is a high-concurrency operation with extremely high requirements for data reliability, and considering that persistent storage will inevitably write a large amount of data to persistent storage space (such as database DB), even a distributed DB will find it difficult to support instantaneous high concurrency. Therefore, the message middleware of a distributed log system can be used to perform asynchronous processing of the submitted questionnaire data based on reliability. That is, the worker node writes the questionnaire data of the target questionnaire to the distributed cache space of the first storage node on the one hand, and writes the questionnaire data of the target questionnaire to the first message middleware queue of the third storage node on the other hand. The third storage node corresponds to the message queue node mentioned above.

[0066] 307. Read the answer data of the target questionnaire from the first message middleware queue, and write the answer data of the target questionnaire into the persistent storage space.

[0067] Specifically, worker nodes can asynchronously read the questionnaire data from the first message middleware queue and write the questionnaire data to the persistent storage space. It can be seen that after the user submits the answer, the backend can first write the questionnaire data to the cache (i.e., the distributed cache space), then send a copy to the distributed log system, and store it in the first message middleware queue for persistent data storage.

[0068] In some feasible implementations, if it is necessary to verify the user's questionnaire data, the questionnaire data can be graded first to obtain the grading results, and the questionnaire data and grading results of the target questionnaire can be written into the distributed cache space of the first storage node and the first message middleware queue of the third storage node.

[0069] In some feasible implementations, the questionnaire data can be encapsulated in a specified format and then written to the first message middleware queue. When a worker node consumes data from the first message middleware queue, that is, when reading questionnaire data from the first message middleware queue, it can first parse the read questionnaire data. If the parsing is successful, the questionnaire data can be written to the answer table in the persistent storage space; if the parsing fails, the questionnaire data can be written as abnormal data to the error table in the persistent storage space. If writing to the answer table fails, or writing to the error table fails, the questionnaire data can be written as abnormal data to the second message middleware queue of the third storage node. The second message middleware queue is different from the first message middleware queue. The second message middleware queue is used to store questionnaire data that failed to be written to the persistent storage space. By introducing an abnormal data queue (i.e., the second message middleware queue), while ensuring that the normal processing flow (i.e., the first message middleware queue) is not blocked, data that failed to be written to the persistent storage space in the first message middleware queue can also be asynchronously written using the second message middleware queue. For example, a worker node can read the questionnaire data from the second message middleware queue and write the questionnaire data into the error table in the persistent storage space, thereby achieving persistent storage of the questionnaire data. The reliability of the data is guaranteed by using the message middleware of the distributed log system and the dual-queue mechanism (including the first message middleware queue for storing normal data and the second message middleware queue for storing abnormal data).

[0070] In some feasible implementations, if writing to the second message middleware queue of the third storage node also fails, it indicates that all types of storage nodes have failed. In this case, the questionnaire data of the target questionnaire can be written to the local disk space to prevent data loss.

[0071] In some feasible implementations, when a questionnaire publisher needs to publish a questionnaire, they can use a client to send a publication request for the target questionnaire to a worker node via the questionnaire submission interface. The publication request carries the publisher's identification information, the target questionnaire's data, and the identification information. The worker node can verify the publisher's identity based on the identification information. After successful verification, it retrieves the updated version number of the target questionnaire from the distributed cache space of the first storage node. For example, a version number auto-increment command can be used to instruct the first storage node to increment the current version number to obtain the updated version number of the target questionnaire. For instance, if the current version number is 3, it can be incremented to 4. The worker node then writes the target questionnaire data to the persistent storage space and the distributed cache space sequentially according to the updated version number. Typically, the concurrency of questionnaire publication is not high because only administrators have the necessary permissions; therefore, the focus in the questionnaire publication process is on data reliability and consistency. Data reliability is ensured through persistent storage and strong synchronization of multiple backups in different locations. This means that questionnaire data can be written to persistent storage spaces on multiple persistent storage nodes. Data consistency is ensured by writing to persistent storage space first and then to distributed cache space. If writing to persistent storage space is successful but writing to cache space fails, the latest data will be written to cache after the cached data expires, ensuring the eventual consistency of the data.

[0072] In this embodiment of the invention, the worker node can respond to the client's request to retrieve the target questionnaire, obtain the version number of the target questionnaire, and prioritize reading the questionnaire data from the local memory space based on the version number. If the target questionnaire does not exist in the local memory space, it will attempt to read the questionnaire data from the distributed cache space or persistent storage space, and write the read target questionnaire data into the local memory space, or into both the local memory space and the distributed cache space, to ensure efficient data reading. After obtaining the user's answer data, the target questionnaire answer data can be written to the distributed cache space of the first storage node and the first message middleware queue of the third storage node in the order of writing. Then, the target questionnaire answer data is read from the first message middleware queue and written into the persistent storage space. This multi-level caching ensures fast response in high-concurrency questionnaire reading scenarios, the version number ensures data timeliness and consistency, and the message middleware queue enables asynchronous data writing.

[0073] In some feasible implementations, such as Figure 4a The above describes a questionnaire publishing process provided by an embodiment of the present invention, which may include:

[0074] Administrators can edit questionnaires through the editing page and submit questionnaire data by calling the CGI (Common Gateway Interface).

[0075] The worker node verifies whether the administrator has owner privileges. If the administrator has the privileges, the questionnaire version number is read incrementally from the distributed cache node. The questionnaire data is then written to the persistent storage space and the distributed cache space in sequence. Data reliability is ensured through persistent storage and strong synchronization of multiple backups in different locations.

[0076] In some feasible implementations, such as Figure 4b The above describes a process for viewing a questionnaire provided by an embodiment of the present invention, which may include:

[0077] The questionnaire participant submits a viewing request by calling the questionnaire CGI file through the questionnaire viewing page.

[0078] The worker node retrieves the questionnaire version number from the distributed cache space. Using the version number, it first checks whether the questionnaire data exists in the service memory. If it exists, it directly returns the questionnaire data; otherwise, it checks whether the questionnaire data exists in the distributed cache space.

[0079] If the questionnaire data exists in the distributed cache, write it to the service memory and return the questionnaire data; if it does not exist in the distributed cache, add a distributed lock to the questionnaire, and then check again whether the questionnaire data exists in the distributed cache. If it exists, release the distributed lock and write the questionnaire data to the service memory; if it still exists, check whether the questionnaire data exists in the persistent storage space.

[0080] If the data exists, the distributed lock is released, and the questionnaire data is written to the distributed cache space and the service memory; if the data does not exist, the distributed lock is released, and an empty data is written to the distributed cache space. This ensures high concurrency through multi-level caching and data reliability through persistent storage and distributed locks.

[0081] In some feasible implementations, such as Figure 4c The above describes a process for submitting a questionnaire according to an embodiment of the present invention, which may include:

[0082] Questionnaire respondents fill out the questionnaire through the page and submit their answers via a CGI function.

[0083] Work nodes can grade the questionnaire data and then write the questionnaire data to the distributed cache space and the message middleware queue of the distributed log system.

[0084] Further, such as Figure 4dThe above describes a process for persistently storing questionnaire data provided in an embodiment of the present invention, which may include:

[0085] Worker nodes consume questionnaire data from the normal message middleware queue of the distributed log system, parse the data, and if the parsing is successful, write it to the answer table in the persistent storage space; if writing to the answer table fails, or parsing fails, write the data to the error table in the persistent storage space; if writing to the error table fails, write the data to the exception message middleware queue of the distributed log system; if writing to the exception message middleware queue fails, write the data to the local disk.

[0086] Further, such as Figure 4e The above describes another process for persistent storage of questionnaire data provided in an embodiment of the present invention, which may include:

[0087] Worker nodes consume abnormal data from the abnormal message middleware queue of the distributed log system and write the data to the error table in the persistent storage space. If writing to the error table fails, an error message is returned, so that the abnormal data can be written to the persistent storage space asynchronously again without blocking the normal processing flow (i.e., the normal message middleware queue of the distributed log system).

[0088] See Figure 5 This is a schematic diagram of a data processing device according to an embodiment of the present invention. The device includes:

[0089] The acquisition module 501 is used to acquire the version number of the target questionnaire in response to the client's questionnaire acquisition request for the target questionnaire.

[0090] The acquisition module 501 is further configured to acquire the questionnaire data of the target questionnaire from a multi-level storage space according to the version number of the target questionnaire, wherein the multi-level storage space includes one or more of local memory space, distributed cache space and persistent storage space.

[0091] The sending module 502 is used to send the questionnaire data of the target questionnaire to the client.

[0092] Optionally, the acquisition module 501 is specifically used for:

[0093] Obtain the identification information of the target questionnaire carried in the questionnaire retrieval request.

[0094] A version retrieval request is sent to the first storage node, the version retrieval request carrying the identification information of the target questionnaire.

[0095] The first storage node receives the version number of the target questionnaire, which is obtained by the first storage node from the distributed cache space based on the identification information of the target questionnaire.

[0096] Optionally, the multi-level storage space includes the local memory space, the distributed cache space, and the persistent storage space, and the acquisition module 501 is specifically used for:

[0097] Based on the version number of the target questionnaire, the questionnaire data is retrieved from the multi-level storage space, including local memory space, distributed cache space, or persistent storage space, according to the set query order.

[0098] Optionally, the acquisition module 501 is specifically used for:

[0099] Based on the version number of the target questionnaire, check whether the local memory space contains the questionnaire data of the target questionnaire.

[0100] If the local memory space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the local memory space according to the version number of the target questionnaire.

[0101] If the local memory space does not contain the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the distributed cache space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the local memory space.

[0102] Optionally, the acquisition module 501 is specifically used for:

[0103] Based on the version number of the target questionnaire, query whether the distributed cache space of the first storage node includes the questionnaire data of the target questionnaire.

[0104] If the distributed cache space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the first storage node is received, and the questionnaire data of the target questionnaire is written into the local memory space.

[0105] If the distributed cache space does not include the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the persistent storage space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the distributed cache space and the local memory space.

[0106] Optionally, the acquisition module 501 is specifically used for:

[0107] Set up a distributed lock for the target questionnaire.

[0108] Based on the version number of the target questionnaire, query whether the persistent storage space of the second storage node includes the questionnaire data of the target questionnaire.

[0109] If the persistent storage space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the second storage node is received, and the distributed lock is released.

[0110] Optionally, the device further includes a processing module 503, wherein:

[0111] The processing module 503 is used to release the distributed lock if the persistent storage space does not contain the questionnaire data of the target questionnaire.

[0112] The sending module 502 is also used to write empty data corresponding to the version number of the target questionnaire into the distributed cache space.

[0113] Optionally, the acquisition module 501 is further configured to query whether the distributed cache space of the first storage node includes the questionnaire data of the target questionnaire based on the version number of the target questionnaire.

[0114] The processing module 503 is further configured to, if the distributed cache space includes the questionnaire data of the target questionnaire, receive the questionnaire data of the target questionnaire sent by the first storage node, release the distributed lock, and write the questionnaire data of the target questionnaire into the local memory space.

[0115] The processing module 503 is further configured to, if the distributed cache space does not include the questionnaire data of the target questionnaire, execute the step of querying whether the persistent storage space of the second storage node includes the questionnaire data of the target questionnaire based on the version number of the target questionnaire.

[0116] Optionally, the processing module 503 is further configured to:

[0117] Receive the answer data of the target questionnaire sent by the client through the answer submission interface.

[0118] The answer data of the target questionnaire is written sequentially to the distributed cache space of the first storage node and the first message middleware queue of the third storage node in the order of writing.

[0119] Read the answer data of the target questionnaire from the first message middleware queue.

[0120] Write the response data of the target questionnaire into the persistent storage space.

[0121] Optionally, the processing module 503 is specifically used for:

[0122] The response data of the target questionnaire is analyzed.

[0123] If the parsing fails, the answer data of the target questionnaire will be written as abnormal data into the error table in the persistent storage space.

[0124] If writing to the error table fails, the answer data of the target questionnaire is written as abnormal data to the second message middleware queue of the third storage node.

[0125] Optionally, the acquisition module 501 is further configured to read the answer data of the target questionnaire from the second message middleware queue.

[0126] The processing module 503 is also used to write the answer data of the target questionnaire into the error table in the persistent storage space.

[0127] Optionally, the processing module 503 is further configured to:

[0128] Receive a publishing request for a target questionnaire sent through the questionnaire submission interface. The publishing request carries the questionnaire publisher's identification information, the questionnaire data of the target questionnaire, and the identification information.

[0129] After the identity of the questionnaire publisher is verified based on the publisher's identification information, the updated version number of the target questionnaire is retrieved from the distributed cache space of the first storage node.

[0130] Based on the updated version number of the target questionnaire, the questionnaire data of the target questionnaire is sequentially written to the persistent storage space of the second storage node and the distributed cache space of the first storage node.

[0131] It should be noted that the functions of each functional module of the data processing device in the embodiments of the present invention can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0132] See Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. The computer device of the present invention includes a power supply module and other structures, and includes a processor 601, a storage device 602, and a network interface 603. The processor 601, the storage device 602, and the network interface 603 can exchange data.

[0133] The storage device 602 may include volatile memory, such as random-access memory (RAM); the storage device 602 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the storage device 602 may also include a combination of the above types of memory.

[0134] The processor 601 may be a central processing unit (CPU). In one embodiment, the processor 601 may also be a graphics processing unit (GPU). The processor 601 may also be a combination of a CPU and a GPU. In one embodiment, the storage device 602 is used to store program instructions. The processor 601 can invoke the program instructions to perform the following operations:

[0135] In response to a client's request to retrieve a target questionnaire, obtain the version number of the target questionnaire.

[0136] The questionnaire data of the target questionnaire is obtained from a multi-level storage space according to the version number of the target questionnaire. The multi-level storage space includes one or more of local memory space, distributed cache space and persistent storage space.

[0137] The questionnaire data of the target questionnaire is sent to the client.

[0138] Optionally, the processor 601 is specifically used for:

[0139] Obtain the identification information of the target questionnaire carried in the questionnaire retrieval request.

[0140] A version retrieval request is sent to the first storage node, the version retrieval request carrying the identification information of the target questionnaire.

[0141] The first storage node receives the version number of the target questionnaire, which is obtained by the first storage node from the distributed cache space based on the identification information of the target questionnaire.

[0142] Optionally, the multi-level storage space includes the local memory space, the distributed cache space, and the persistent storage space, and the processor 601 is specifically used for:

[0143] Based on the version number of the target questionnaire, the questionnaire data is retrieved from the multi-level storage space, including local memory space, distributed cache space, or persistent storage space, according to the set query order.

[0144] Optionally, the processor 601 is specifically used for:

[0145] Based on the version number of the target questionnaire, check whether the local memory space contains the questionnaire data of the target questionnaire.

[0146] If the local memory space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the local memory space according to the version number of the target questionnaire.

[0147] If the local memory space does not contain the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the distributed cache space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the local memory space.

[0148] Optionally, the processor 601 is specifically used for:

[0149] Based on the version number of the target questionnaire, query whether the distributed cache space of the first storage node includes the questionnaire data of the target questionnaire.

[0150] If the distributed cache space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the first storage node is received, and the questionnaire data of the target questionnaire is written into the local memory space.

[0151] If the distributed cache space does not include the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the persistent storage space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the distributed cache space and the local memory space.

[0152] Optionally, the processor 601 is specifically used for:

[0153] Set up a distributed lock for the target questionnaire.

[0154] Based on the version number of the target questionnaire, query whether the persistent storage space of the second storage node includes the questionnaire data of the target questionnaire.

[0155] If the persistent storage space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the second storage node is received, and the distributed lock is released.

[0156] Optionally, the processor 601 is further configured to:

[0157] If the persistent storage space does not contain the questionnaire data of the target questionnaire, then the distributed lock is released.

[0158] Write empty data corresponding to the version number of the target questionnaire into the distributed cache space.

[0159] Optionally, the processor 601 is further configured to:

[0160] Based on the version number of the target questionnaire, query whether the distributed cache space of the first storage node includes the questionnaire data of the target questionnaire.

[0161] If the distributed cache space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the first storage node is received, the distributed lock is released, and the questionnaire data of the target questionnaire is written into the local memory space.

[0162] If the distributed cache space does not include the questionnaire data of the target questionnaire, then the step of querying whether the persistent storage space of the second storage node includes the questionnaire data of the target questionnaire based on the version number of the target questionnaire is executed.

[0163] Optionally, the processor 601 is further configured to:

[0164] Receive the answer data of the target questionnaire sent by the client through the answer submission interface.

[0165] The answer data of the target questionnaire is written sequentially to the distributed cache space of the first storage node and the first message middleware queue of the third storage node in the order of writing.

[0166] Read the answer data of the target questionnaire from the first message middleware queue.

[0167] Write the response data of the target questionnaire into the persistent storage space.

[0168] Optionally, the processor 601 is specifically used for:

[0169] The response data of the target questionnaire is analyzed.

[0170] If the parsing fails, the answer data of the target questionnaire will be written as abnormal data into the error table in the persistent storage space.

[0171] If writing to the error table fails, the answer data of the target questionnaire is written as abnormal data to the second message middleware queue of the third storage node.

[0172] Optionally, the processor 601 is further configured to:

[0173] Read the response data of the target questionnaire from the second message middleware queue.

[0174] Write the answer data of the target questionnaire into the error table in the persistent storage space.

[0175] Optionally, the processor 601 is further configured to:

[0176] Receive a publishing request for a target questionnaire sent through the questionnaire submission interface. The publishing request carries the questionnaire publisher's identification information, the questionnaire data of the target questionnaire, and the identification information.

[0177] After the identity of the questionnaire publisher is verified based on the publisher's identification information, the updated version number of the target questionnaire is retrieved from the distributed cache space of the first storage node.

[0178] Based on the updated version number of the target questionnaire, the questionnaire data of the target questionnaire is sequentially written to the persistent storage space of the second storage node and the distributed cache space of the first storage node.

[0179] In specific implementations, the processor 601, storage device 602, and network interface 603 described in the embodiments of the present invention can execute the embodiments of the present invention. Figure 2 or Figure 3 The implementation described in the relevant embodiments of the provided data processing method can also be used to execute the embodiments of the present invention. Figure 5 The implementation methods described in the relevant embodiments of the provided data processing device will not be repeated here.

[0180] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program includes one or more instructions and can be stored in a computer storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0181] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps described in the embodiments of the above methods.

[0182] The above-disclosed embodiments are only some of the embodiments of this application, and should not be construed as limiting the scope of this application. Therefore, any equivalent changes made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: In response to a client's request to retrieve a target questionnaire, obtain the version number of the target questionnaire; The questionnaire data of the target questionnaire is obtained from a multi-level storage space according to the version number of the target questionnaire. The multi-level storage space includes at least two of the following: local memory space, distributed cache space, and persistent storage space. Send the questionnaire data of the target questionnaire to the client; Receive the answer data of the target questionnaire sent by the client through the answer submission interface; The answer data of the target questionnaire is written sequentially to the distributed cache space of the first storage node and the first message middleware queue of the third storage node in the order of writing. Read the answer data of the target questionnaire from the first message middleware queue, and parse the answer data of the target questionnaire; If the parsing fails, the answer data of the target questionnaire will be written as abnormal data into the error table in the persistent storage space. If writing to the error table fails, the answer data of the target questionnaire is written as abnormal data to the second message middleware queue of the third storage node; the answer data of the target questionnaire is read from the second message middleware queue and written to the error table in the persistent storage space.

2. The method according to claim 1, characterized in that, Obtaining the version number of the target questionnaire includes: Obtain the identification information of the target questionnaire carried in the questionnaire retrieval request; Send a version retrieval request to the first storage node, the version retrieval request carrying the identification information of the target questionnaire; The first storage node receives the version number of the target questionnaire, which is obtained by the first storage node from the distributed cache space based on the identification information of the target questionnaire.

3. The method according to claim 1 or 2, characterized in that, The multi-level storage space includes the local memory space, the distributed cache space, and the persistent storage space. The step of retrieving the questionnaire data of the target questionnaire from the multi-level storage space according to the version number of the target questionnaire includes: Based on the version number of the target questionnaire, the questionnaire data is retrieved from the multi-level storage space, including local memory space, distributed cache space, or persistent storage space, according to the set query order.

4. The method according to claim 3, characterized in that The step of retrieving the questionnaire data of the target questionnaire from the multi-level storage space, including local memory space, distributed cache space, or persistent storage space, according to the version number of the target questionnaire and in a set query order includes: Based on the version number of the target questionnaire, query whether the local memory space contains the questionnaire data of the target questionnaire; If the local memory space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the local memory space according to the version number of the target questionnaire; If the local memory space does not contain the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the distributed cache space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the local memory space.

5. The method according to claim 4, characterized in that, The step of reading the questionnaire data of the target questionnaire from the distributed cache space according to the version number of the target questionnaire, and writing the questionnaire data of the target questionnaire into the local memory space includes: Based on the version number of the target questionnaire, query whether the distributed cache space of the first storage node includes the questionnaire data of the target questionnaire; If the distributed cache space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the first storage node is received, and the questionnaire data of the target questionnaire is written into the local memory space. If the distributed cache space does not include the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire is read from the persistent storage space according to the version number of the target questionnaire, and the questionnaire data of the target questionnaire is written into the distributed cache space and the local memory space.

6. The method according to claim 5, characterized in that, The step of reading the questionnaire data of the target questionnaire from the persistent storage space according to the version number of the target questionnaire includes: Set a distributed lock for the target questionnaire; Based on the version number of the target questionnaire, query whether the persistent storage space of the second storage node includes the questionnaire data of the target questionnaire; If the persistent storage space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the second storage node is received, and the distributed lock is released.

7. The method according to claim 6, characterized in that, The method further includes: If the persistent storage space does not include the questionnaire data of the target questionnaire, then the distributed lock is released, and empty data corresponding to the version number of the target questionnaire is written to the distributed cache space.

8. The method according to claim 6 or 7, characterized in that, After setting a distributed lock for the target questionnaire, and before querying whether the persistent storage space of the persistent storage node includes the questionnaire data of the target questionnaire based on the version number of the target questionnaire, the method further includes: Based on the version number of the target questionnaire, query whether the distributed cache space of the first storage node includes the questionnaire data of the target questionnaire; If the distributed cache space includes the questionnaire data of the target questionnaire, then the questionnaire data of the target questionnaire sent by the first storage node is received, the distributed lock is released, and the questionnaire data of the target questionnaire is written into the local memory space. If the distributed cache space does not include the questionnaire data of the target questionnaire, then the step of querying whether the persistent storage space of the second storage node includes the questionnaire data of the target questionnaire based on the version number of the target questionnaire is executed.

9. The method according to claim 1 or 2, characterized in that, The method further includes: Receive a publishing request for a target questionnaire sent through the questionnaire submission interface. The publishing request carries the questionnaire publisher's identification information, the questionnaire data of the target questionnaire, and the identification information. After the identity of the questionnaire publisher is verified based on the identifier information of the questionnaire publisher, the updated version number of the target questionnaire is obtained from the distributed cache space of the first storage node; Based on the updated version number of the target questionnaire, the questionnaire data of the target questionnaire is sequentially written to the persistent storage space of the second storage node and the distributed cache space of the first storage node.

10. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to obtain the version number of the target questionnaire in response to the client's questionnaire acquisition request for the target questionnaire; The acquisition module is further configured to acquire the questionnaire data of the target questionnaire from a multi-level storage space according to the version number of the target questionnaire, wherein the multi-level storage space includes at least two of the following: local memory space, distributed cache space, and persistent storage space. The sending module is used to send the questionnaire data of the target questionnaire to the client; The processing module is configured to receive the answer data of the target questionnaire sent by the client through the answer submission interface; sequentially write the answer data of the target questionnaire into the distributed cache space of the first storage node and the first message middleware queue of the third storage node in the writing order; read the answer data of the target questionnaire from the first message middleware queue and parse the answer data of the target questionnaire; if the parsing fails, write the answer data of the target questionnaire as abnormal data into the error table in the persistent storage space; if writing to the error table fails, write the answer data of the target questionnaire as abnormal data into the second message middleware queue of the third storage node. The acquisition module is also used to read the answer data of the target questionnaire from the second message middleware queue; The processing module is also used to write the answer data of the target questionnaire into the error table in the persistent storage space.

11. A computer device, characterized in that, The computer device includes a processor, a network interface, and a storage device, which are interconnected. The network interface is controlled by the processor to send and receive data. The storage device is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute the data processing method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that are executed by a processor to perform the data processing method as described in any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the data processing method as described in any one of claims 1-9.

Images