Data processing method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides a data processing method, device, electronic device, and storage medium, comprising: obtaining a keyword and a first data version number of a batch of data to be processed sent by a client; for each piece of data to be processed, querying a second data version number corresponding to the keyword from a cache database stored on a solid-state hard drive; comparing the queried second data version number with the first data version number, and filtering the batch of data to be processed based on the comparison result; updating the data stored in a background database based on the filtered data to be processed, and using the updated data in the background database for task processing. The present invention uses a database stored on a solid-state hard drive for data query, and stores the keyword and data version number fields of the data on the solid-state hard drive. Due to the large storage capacity of the solid-state hard drive, the memory occupancy rate can be reduced, thereby reducing storage costs.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, a data processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of smart terminals, videos can be watched through video clients on smart terminals. Users can post comments and barrages on video content, and these data can be delivered to the background for processing through message queues.

[0003] Because redundant data is easily generated due to factors such as network latency, it is often necessary to query the data stored in memory to intercept redundant data in the delivered data. However, in high-concurrency scenarios, large batches of data are delivered to the backend, so a large amount of data needs to be stored in memory for query purposes, which can easily lead to high memory usage and increased storage costs. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a data processing method, a data processing device, an electronic device, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0005] In order to solve the above problems, an embodiment of the present invention discloses a data processing method, which includes:

[0006] Obtain the keyword and first data version number of the batch of data to be processed sent by the client;

[0007] For each piece of data to be processed, query a second data version number corresponding to the keyword from a cache database stored on a solid-state drive;

[0008] Comparing the retrieved second data version number with the first data version number, and filtering the batch of data to be processed based on the comparison result;

[0009] The data stored in the background database is updated according to the data to be processed obtained by screening, and the updated data in the background database is used for task processing.

[0010] Optionally, screening the batch of data to be processed based on the comparison result includes:

[0011] If the queried second data version number is greater than the first data version number, discard the data to be processed;

[0012] If the queried second data version number is less than or equal to the first data version number, the data to be processed is retained, and the second data version number recorded in the cache database is updated to the first data version number.

[0013] Optionally, after querying the second data version number corresponding to the keyword, the method further includes:

[0014] If the second data version number corresponding to the keyword is not found, the data to be processed is stored in the cache database.

[0015] Optionally, updating the data stored in the background database based on the filtered data to be processed includes:

[0016] Perform business logic processing on the filtered data to be processed;

[0017] Update the data stored in the background database based on the data processed by business logic.

[0018] Optionally, updating the data in the backend database according to the data processed by the business logic includes:

[0019] For each piece of data processed by the business logic, query the backend database for a third data version number corresponding to the keyword of the data processed by the business logic;

[0020] The queried third data version number is compared with the first data version number of the data processed by the business logic, and based on the comparison result, the data corresponding to the keyword of the data processed by the business logic in the background database is updated.

[0021] Optionally, updating data corresponding to the keywords of the data processed by the business logic in the background database based on the comparison result includes:

[0022] If the queried third data version number is less than or equal to the first data version number of the data processed by the business logic, the data processed by the business logic is used to update the data corresponding to the keyword of the data processed by the business logic in the background database;

[0023] If the third data version number found is greater than the first data version number of the data processed by the business logic, the data in the background database remains unchanged.

[0024] Optionally, obtaining the keyword and the first data version number of the user behavior data sent by the client includes:

[0025] Monitor the batch of data to be processed sent by the client;

[0026] The batch of data to be processed is parsed to obtain a keyword and a first data version number of the batch of data to be processed.

[0027] An embodiment of the present invention further discloses a data processing device, comprising:

[0028] An acquisition module, configured to acquire a keyword and a first data version number of the batch of data to be processed sent by the client;

[0029] A query module, configured to query, for each piece of data to be processed, a second data version number corresponding to the keyword from a cache database stored on a solid-state drive;

[0030] a comparison module, configured to compare the queried second data version number with the first data version number, and filter the batch of data to be processed based on the comparison result;

[0031] The updating module is used to update the data stored in the background database according to the data to be processed obtained by screening, and the updated data in the background database is used for task processing.

[0032] Optionally, the comparison module includes:

[0033] a discarding submodule, configured to discard the data to be processed if the queried second data version number is greater than the first data version number;

[0034] The retaining submodule is configured to retain the data to be processed and update the second data version number recorded in the cache database to the first data version number if the queried second data version number is less than or equal to the first data version number.

[0035] Optionally, after querying the second data version number corresponding to the keyword, the method further includes:

[0036] The insertion module is configured to store the data to be processed into the cache database if the second data version number corresponding to the keyword is not found.

[0037] Optionally, the update module includes:

[0038] The business logic processing submodule is used to perform business logic processing on the filtered data to be processed;

[0039] The data update submodule is used to update the data stored in the background database according to the data processed by the business logic.

[0040] Optionally, the data updating submodule includes:

[0041] A version number query unit, configured to query, for each piece of data processed by the business logic, a third data version number corresponding to a keyword of the data processed by the business logic from a background database;

[0042] The version number comparison unit is used to compare the queried third data version number with the first data version number of the data processed by the business logic, and based on the comparison result, update the data corresponding to the keyword of the data processed by the business logic in the background database.

[0043] Optionally, the version number comparison unit includes:

[0044] a background data updating subunit, configured to update data corresponding to a keyword of the data processed by the business logic in a background database using the data processed by the business logic if the third data version number queried is less than or equal to the first data version number of the data processed by the business logic;

[0045] The background data non-update subunit is used to keep the data in the background database unchanged if the third data version number queried is greater than the first data version number of the data processed by the business logic.

[0046] Optionally, the acquisition module includes:

[0047] The monitoring submodule is used to monitor the batch data to be processed sent by the client;

[0048] The parsing submodule is used to parse the batch of data to be processed to obtain a keyword and a first data version number of the batch of data to be processed.

[0049] An embodiment of the present invention further discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the above-mentioned data processing method when executed by the processor.

[0050] An embodiment of the present invention further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0051] The embodiments of the present invention include the following advantages:

[0052] In an embodiment of the present invention, a keyword and a first data version number of a batch of to-be-processed data sent by a client are obtained; for each piece of to-be-processed data, a second data version number corresponding to the keyword is queried from a cache database based on solid-state hard disk storage; the queried second data version number is compared with the first data version number, and the batch of to-be-processed data is filtered based on the comparison result; based on the filtered to-be-processed data, the data stored in the background database is updated, and the updated data in the background database is used for task processing. Compared with the prior art that uses a database based on memory storage for data query, the present invention uses a database based on solid-state hard disk storage for data query, and stores the keyword and data version number fields of the data on the solid-state hard disk. Due to the large storage capacity of the solid-state hard disk, the memory occupancy rate can be reduced, thereby reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of a data processing method provided by an embodiment of the present invention;

[0054] Figure 2 is a flowchart of another data processing method provided by an embodiment of the present invention;

[0055] Figure 2A-2E is a flow chart of sub-steps of another data processing method provided by an embodiment of the present invention;

[0056] Figure 3 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0057] Figure 4 This is a structural block diagram of a data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Data delivered to the backend often contains redundant data, requiring a query of the data stored in memory to intercept any redundant data. However, in high-concurrency scenarios, large volumes of data are delivered to the backend, requiring a large amount of data to be stored in memory for querying. This can lead to high memory usage and, in turn, increased memory costs.

[0060] One of the core concepts of an embodiment of the present invention is to obtain the keyword and first data version number of the batch of data to be processed sent by the client; for each piece of data to be processed, query the second data version number corresponding to the keyword from the cache database based on solid-state hard disk storage; compare the queried second data version number with the first data version number, and filter the batch of data to be processed based on the comparison result; according to the filtered data to be processed, update the data stored in the background database, and the updated data in the background database is used for task processing. Compared with the prior art that uses a database based on memory storage for data query, the present invention uses a database based on solid-state hard disk storage for data query, and stores the keyword and data version number fields of the data on the solid-state hard disk. Due to the large storage capacity of the solid-state hard disk, the memory occupancy rate can be reduced, thereby reducing storage costs.

[0061] Reference Figure 1 , shows a flowchart of a data processing method provided by an embodiment of the present invention, the method may specifically include the following steps:

[0062] Step 101: Obtain a keyword and a first data version number of the batch of to-be-processed data sent by the client.

[0063] The data processing method of an embodiment of the present invention can be applied to task processing scenarios. For example, users can watch videos through a video client. When a large number of users need to post barrages and comments on the videos they are watching, a large amount of data to be processed will be generated. These data to be processed can be delivered to the content security review platform through a message queue, and the data version number recorded in the cache database based on solid-state hard drive storage can be queried. The data version number of each piece of data to be processed is compared with the queried data version number, and the batch of data to be processed is filtered based on the comparison results. The data stored in the background database is updated based on the filtered data to be processed, so that the content security review platform can perform content security review on the updated data.

[0064] In actual applications, the data to be processed can also be delivered to the customer service platform to perform task processing on the updated data; the data to be processed can also be delivered to the OA (Office Automation) management platform to perform OA task work order processing on the updated data, etc. The present invention is not limited here.

[0065] A keyword can refer to a data identifier, a single field, or a combination of multiple fields. A data version number can refer to a numerical value used to indicate the number of times data has been updated. For example, when a user enters a comment, due to network delays and other reasons, the same comment data may be delivered twice. The data version number of the comment data delivered the first time may be 1, and the data version number of the comment data delivered the second time may be 2. For another example, a user can modify a comment that has been published. The data version number of the comment data before the modification may be 1, and the data version number of the modified comment data may be 2.

[0066] Step 102: for each piece of data to be processed, query a second data version number corresponding to the keyword from a cache database stored on a solid state drive.

[0067] The cache database can be a database based on a solid-state drive (SSD) that stores data and can be used to cache data keywords and data version number fields. In an embodiment of the present invention, a database based on a SSD that stores data is used for querying. Due to the large storage capacity of SSDs, in high-concurrency scenarios, by storing the data keywords and data version number fields on the SSD, memory usage can be reduced, thereby reducing storage costs.

[0068] Step 103: compare the retrieved second data version number with the first data version number, and filter the batch of data to be processed based on the comparison result.

[0069] For each piece of data to be processed, the second data version number retrieved is compared with the first data version number of the data to be processed. Based on the comparison result, it is possible to determine whether the data to be processed is redundant, thereby filtering out redundant data in the batch of data to be processed. In this embodiment of the present invention, by filtering the batch of data to be processed, it is possible to avoid the problem of excessive storage pressure on the backend database caused by excessive redundant data, which can waste computing resources and increase computing pressure when performing task processing.

[0070] Step 104 : updating the data in the background database based on the filtered data to be processed. The updated data in the background database is used for task processing.

[0071] After completing the data screening, the data stored in the background database can be updated, so that task processing can be performed based on the updated data.

[0072] In an embodiment of the present invention, a keyword and a first data version number of a batch of to-be-processed data sent by a client are obtained; for each piece of to-be-processed data, a second data version number corresponding to the keyword is queried from a cache database based on solid-state hard disk storage; the queried second data version number is compared with the first data version number, and the batch of to-be-processed data is filtered based on the comparison result; based on the filtered to-be-processed data, the data stored in the background database is updated, and the updated data in the background database is used for task processing. Compared with the prior art that uses a database based on memory storage for data query, the present invention uses a database based on solid-state hard disk storage for data query, and stores the keyword and data version number fields of the data on the solid-state hard disk. Due to the large storage capacity of the solid-state hard disk, the memory occupancy rate can be reduced, thereby reducing storage costs.

[0073] Reference Figure 2 , shows a flowchart of a data processing method provided by an embodiment of the present invention, the method may specifically include the following steps:

[0074] Step 201: Obtain the keyword and first data version number of the batch of to-be-processed data sent by the client.

[0075] In an alternative embodiment, referring to Figure 2A , the step 201 may include the following sub-steps S11-S12:

[0076] Sub-step S11, monitoring the batch of to-be-processed data sent by the client.

[0077] Sub-step S12: parsing the batch of data to be processed to obtain a keyword and a first data version number of the batch of data to be processed.

[0078] In an embodiment of the present invention, the data stream sent by the client can be monitored. The data stream may include a large amount of data to be processed. Each piece of data to be processed can be parsed to obtain a unique keyword key and a first data version number.

[0079] Step 202: for each piece of data to be processed, query the cache database based on the solid state drive for a second data version number corresponding to the keyword.

[0080] Step 203: If the second data version number corresponding to the keyword is not found, the data to be processed is stored in the cache database.

[0081] When querying each pending data item, if the second data version number corresponding to the keyword key is not found in the cache database stored on the solid-state drive (SSD), that is, the data version number corresponding to the keyword does not exist in the cache database, the pending data item can be retained and added to the cache database. In a business scenario, when a user posts a comment for the first time, the corresponding pending data item is generated. The corresponding second data version number cannot be found in the cache database, so the pending data item can be added to the cache database for storage.

[0082] Step 204: compare the retrieved second data version number with the first data version number, and filter the batch of data to be processed based on the comparison result.

[0083] In an alternative embodiment, referring to Figure 2B The step of screening the batch of data to be processed based on the comparison results in step 204 may include the following sub-steps S21-S22:

[0084] Sub-step S21: If the queried second data version number is greater than the first data version number, the data to be processed is discarded.

[0085] Sub-step S22: If the queried second data version number is less than or equal to the first data version number, retain the data to be processed, and update the second data version number recorded in the cache database to the first data version number.

[0086] The data version number may refer to a numerical value used to indicate the number of times the data has been updated. The larger the data version number, the later the data was generated and the newer the data. When the second data version number queried is greater than the first data version number, it means that the data being delivered is expired data, and the data to be processed is discarded, thereby filtering out expired data in the data stream. When the second data version number queried is equal to the first data version number, it means that the data being delivered is duplicate data, and the data recorded in the cache database may be updated, and the data version number field and other fields of the data may be updated to the cache database. When the second data version number queried is less than the first data version number, it means that the data being delivered is new data, and the data recorded in the cache database may be updated, and the data version number field and other fields of the data may be updated to the cache database.

[0087] Step 205 : updating the data stored in the background database based on the filtered data to be processed. The updated data in the background database is used for task processing.

[0088] In an alternative embodiment, referring to Figure 2CThe step of updating the data stored in the background database according to the filtered data to be processed in step 205 may include the following sub-steps S31-S32:

[0089] Sub-step S31, performing business logic processing on the filtered data to be processed.

[0090] Sub-step S32: updating the data stored in the background database according to the data processed by the business logic.

[0091] After the data to be processed is filtered, business logic processing such as filtering sensitive words and anti-spam can be performed on the filtered data to be processed, so as to update the data stored in the background database based on the data processed by the business logic.

[0092] In an alternative embodiment, referring to Figure 2D , the sub-step S32 may include the following sub-steps S321-S322:

[0093] Sub-step S321 : for each piece of data processed by the business logic, query the backend database for a third data version number corresponding to the keyword of the data processed by the business logic.

[0094] Sub-step S322, compares the queried third data version number with the first data version number of the data processed by the business logic, and updates the data corresponding to the keyword of the data processed by the business logic in the background database based on the comparison result.

[0095] After performing business logic processing on the data, for each piece of data, the third data version number corresponding to the keyword of the data can be queried from the background database, and the third data version number of the data can be compared with the first data version number, and the data in the background database can be updated based on the comparison result.

[0096] In an alternative embodiment, referring to Figure 2E The step of updating the data corresponding to the keyword of the data processed by the business logic in the background database based on the comparison result in sub-step S322 may include the following sub-steps S3221-S3222:

[0097] Sub-step S3221, if the queried third data version number is less than or equal to the first data version number of the data processed by the business logic, use the data processed by the business logic to update the data corresponding to the keyword of the data processed by the business logic in the background database.

[0098] Sub-step S3222: If the third data version number found is greater than the first data version number of the data processed by the business logic, the data in the background database is kept unchanged.

[0099] If the third data version number corresponding to the data keyword cannot be found in the backend database, or the third data version number found is less than or equal to the first data version number of the data processed by the business logic, it indicates that the delivered data is new data that needs to be updated and saved in the backend database, and the data can be updated in the backend database. If the third data version number found is greater than the first data version number of the data processed by the business logic, it indicates that the delivered data is expired, and the data will not be updated in the backend database.

[0100] In an embodiment of the present invention, a data version number recording mechanism is adopted to implement optimistic locking. In the prior art, when filtering large batches of data to be processed, a pessimistic locking mechanism is usually adopted to improve query efficiency due to the large amount of redundant data therein. The present invention screens the batches of data to be processed by comparing the second data version number in the cache database with the first data version number of the data to be processed before updating the data to the background database. Therefore, when updating the data to the background database, the amount of redundant data is greatly reduced, and the optimistic locking mechanism can improve query efficiency.

[0101] After updating the backend data, you can continue to update other storage modules, such as the data lake and elasticsearch index.

[0102] In an embodiment of the present invention, a keyword and a first data version number of a batch of to-be-processed data sent by a client are obtained; for each piece of to-be-processed data, a second data version number corresponding to the keyword is queried from a cache database based on solid-state hard disk storage; the queried second data version number is compared with the first data version number, and the batch of to-be-processed data is filtered based on the comparison result; based on the filtered to-be-processed data, the data stored in the background database is updated, and the updated data in the background database is used for task processing. Compared with the prior art that uses a database based on memory storage for data query, the present invention uses a database based on solid-state hard disk storage for data query, and stores the keyword and data version number fields of the data on the solid-state hard disk. Due to the large storage capacity of the solid-state hard disk, the memory occupancy rate can be reduced, thereby reducing storage costs.

[0103] In order to enable those skilled in the art to better understand the embodiments of the present invention, the embodiments of the present invention are described below. Figure 3 , which is a flowchart of a data processing method provided by an embodiment of the present invention.

[0104] S1, process starts;

[0105] S2, monitor the data stream that needs to be processed;

[0106] S3. Parse the data to obtain the data keyword and the first data version number;

[0107] S4. Query the cache database for a second data version number corresponding to the keyword, and compare the second data version number with the first data version number;

[0108] S5. If the second data version number cannot be found, that is, the second data version does not exist in the cache database, then the data is added to the cache database;

[0109] S6. If the second data version number found is greater than the first data version number, discard the data;

[0110] S7. If the second data version number found is less than or equal to the first data version number, update the data corresponding to the data keyword in the cache database;

[0111] S8. Perform business logic processing on the filtered data;

[0112] S9. Query the backend database for a third data version number corresponding to the keyword of the data processed by the business logic, compare the third data version number with the first data version number, and update the data based on the comparison result, so as to update the data based on the optimistic locking mechanism;

[0113] S10. Update data in other storage modules such as the data lake;

[0114] S11. The process ends.

[0115] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0116] Reference Figure 4 , shows a structural block diagram of a data processing device provided by an embodiment of the present invention, which may specifically include the following modules:

[0117] An acquisition module 401 is configured to acquire a keyword and a first data version number of the batch of data to be processed sent by the client;

[0118] A query module 402 is configured to query, for each piece of data to be processed, a second data version number corresponding to the keyword from a cache database stored on a solid-state drive;

[0119] A comparison module 403 is configured to compare the retrieved second data version number with the first data version number, and filter the batch of data to be processed based on the comparison result;

[0120] The updating module 404 is used to update the data stored in the background database according to the filtered data to be processed. The updated data in the background database is used for task processing.

[0121] In an optional embodiment, the comparison module includes:

[0122] a discarding submodule, configured to discard the data to be processed if the queried second data version number is greater than the first data version number;

[0123] The retaining submodule is configured to retain the data to be processed and update the second data version number recorded in the cache database to the first data version number if the queried second data version number is less than or equal to the first data version number.

[0124] In an optional embodiment, after querying the second data version number corresponding to the keyword, the method further includes:

[0125] The insertion module is configured to store the data to be processed into the cache database if the second data version number corresponding to the keyword is not found.

[0126] In an optional embodiment, the update module includes:

[0127] The business logic processing submodule is used to perform business logic processing on the filtered data to be processed;

[0128] The data update submodule is used to update the data stored in the background database according to the data processed by the business logic.

[0129] In an optional embodiment, the data updating submodule includes:

[0130] A version number query unit, configured to query, for each piece of data processed by the business logic, a third data version number corresponding to a keyword of the data processed by the business logic from a background database;

[0131] The version number comparison unit is used to compare the queried third data version number with the first data version number of the data processed by the business logic, and based on the comparison result, update the data corresponding to the keyword of the data processed by the business logic in the background database.

[0132] In an optional embodiment, the version number comparison unit includes:

[0133] a background data updating subunit, configured to update data corresponding to a keyword of the data processed by the business logic in a background database using the data processed by the business logic if the third data version number queried is less than or equal to the first data version number of the data processed by the business logic;

[0134] The background data non-update subunit is used to keep the data in the background database unchanged if the third data version number queried is greater than the first data version number of the data processed by the business logic.

[0135] In an optional embodiment, the acquisition module includes:

[0136] The monitoring submodule is used to monitor the batch data to be processed sent by the client;

[0137] The parsing submodule is used to parse the batch of data to be processed to obtain a keyword and a first data version number of the batch of data to be processed.

[0138] In an embodiment of the present invention, a keyword and a first data version number of a batch of to-be-processed data sent by a client are obtained; for each piece of to-be-processed data, a second data version number corresponding to the keyword is queried from a cache database based on solid-state hard disk storage; the queried second data version number is compared with the first data version number, and the batch of to-be-processed data is filtered based on the comparison result; based on the filtered to-be-processed data, the data stored in the background database is updated, and the updated data in the background database is used for task processing. Compared with the prior art that uses a database based on memory storage for data query, the present invention uses a database based on solid-state hard disk storage for data query, and stores the keyword and data version number fields of the data on the solid-state hard disk. Due to the large storage capacity of the solid-state hard disk, the memory occupancy rate can be reduced, thereby reducing storage costs.

[0139] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] An embodiment of the present invention further provides an electronic device, including:

[0141] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0142] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0143] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0144] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0148] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0149] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0150] The data processing method, device, electronic device and storage medium provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain the keyword and first data version number of the batch of data to be processed sent by the client; For each piece of data to be processed, query a second data version number corresponding to the keyword from a cache database stored on a solid-state drive; Comparing the retrieved second data version number with the first data version number, and filtering the batch of data to be processed based on the comparison result; The data stored in the background database is updated according to the data to be processed obtained by screening, and the updated data in the background database is used for task processing.

2. The method according to claim 1, characterized in that The screening of the batch of data to be processed based on the comparison result includes: If the queried second data version number is greater than the first data version number, discard the data to be processed; If the queried second data version number is less than or equal to the first data version number, the data to be processed is retained, and the second data version number recorded in the cache database is updated to the first data version number.

3. The method according to claim 1, characterized in that After querying the second data version number corresponding to the keyword, the method further includes: If the second data version number corresponding to the keyword is not found, the data to be processed is stored in the cache database.

4. The method according to claim 1, wherein The updating of the data stored in the background database according to the data to be processed obtained by screening includes: Perform business logic processing on the filtered data to be processed; Update the data stored in the background database based on the data processed by business logic.

5. The method according to claim 4, characterized in that The data stored in the backend database is updated according to the data processed by the business logic, including: For each piece of data processed by the business logic, query the backend database for a third data version number corresponding to the keyword of the data processed by the business logic; The queried third data version number is compared with the first data version number of the data processed by the business logic, and based on the comparison result, the data corresponding to the keyword of the data processed by the business logic in the background database is updated.

6. The method according to claim 5, characterized in that The updating of data corresponding to the keywords of the data processed by the business logic in the background database based on the comparison result includes: If the queried third data version number is less than or equal to the first data version number of the data processed by the business logic, the data processed by the business logic is used to update the data corresponding to the keyword of the data processed by the business logic in the background database; If the third data version number found is greater than the first data version number of the data processed by the business logic, the data in the background database remains unchanged.

7. The method according to claim 1, characterized in that The step of obtaining the keyword and the first data version number of the user behavior data sent by the client includes: Monitor the batch of data to be processed sent by the client; The batch of data to be processed is parsed to obtain a keyword and a first data version number of the batch of data to be processed.

8. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a keyword and a first data version number of the batch of data to be processed sent by the client; A query module, configured to query, for each piece of data to be processed, a second data version number corresponding to the keyword from a cache database stored on a solid-state drive; a comparison module, configured to compare the queried second data version number with the first data version number, and filter the batch of data to be processed based on the comparison result; The updating module is used to update the data stored in the background database according to the data to be processed obtained by screening, and the updated data in the background database is used for task processing.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images