Data processing method and device, computer equipment and storage medium
By receiving data requests on the write side in the distributed database, determining the processing method based on the data description information, pushing and processing unwritten data to avoid loss, solving the data loss problem in the distributed database and improving data throughput.
Patent Information
- Application Number
- CN202411386270.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-03
AI Technical Summary
In a distributed database, data loss problem caused by delays between the write side and the read side, especially when the delay between sending a data request on the write side and the data is actually written to the database is too long, the read side Data that has not been written may not be read.
By receiving the data write request sent by the write side, the write judgment conditions are determined based on the data description information, and the data processing method is determined based on the conditions. If the write processing conditions are not met, the push processing is adopted, and the data is directly pushed to the reading side , to avoid data loss, and at the same time, it supports multiple terminal devices on the write side to perform write tasks concurrently.
It effectively solves the problem of data loss due to delay and improves data throughput. Compared with traditional methods such as distributed locking, it provides a more efficient data processing solution.
Smart Images
Figure CN119248803A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art
[0002] Existing distributed databases can provide users with data storage functions. Data can be stored in a table storage system. The write side can write data to the table storage system, and the read side can read data from the table storage system.
[0003] In some business scenarios, the reading side (also called the consumer side) is generally required to ensure that each row of data written is read at least once, that is, missed data is unacceptable. However, a distributed database is a cluster composed of several unrelated machines, and the writing side and the reading side of the data are not a single point, that is, the writing side and the reading side themselves are also distributed. In the actual application process, the following problems will arise: When the writing side writes data, there will be a delay between sending a write request to the distributed database and the data actually being written to the database (such as network congestion, thread debugging, etc.). If during this period, the reading side reads the data in the database, but because the row of data has not been written, the row of data cannot be read, a situation similar to data loss occurs. Summary of the invention
[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment and storage medium that can effectively avoid data loss in response to the above technical problems.
[0005] In a first aspect, the present application provides a data processing method. The method comprises:
[0006] Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written;
[0007] Determine the writing judgment condition of the data to be written according to the data description information of the data to be written;
[0008] Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0009] The data to be written is processed according to the data processing method.
[0010] In one embodiment, determining the writing judgment condition of the data to be written according to the data description information of the data to be written includes:
[0011] If the data description information indicates that the data to be written includes sequence data, determining that the writing judgment condition of the data to be written is a numbering judgment condition;
[0012] If the data description information indicates that the data to be written does not include sequence data, it is determined that the writing judgment condition of the data to be written is a time judgment condition.
[0013] In one embodiment, determining a data processing method for data to be written according to a write judgment condition and a data write request includes:
[0014] In the case where the write judgment condition is a time judgment condition, a data processing method for the data to be written is determined according to the current reference time corresponding to the time judgment condition and the data write request.
[0015] In one of the embodiments, the data write request also carries an expected read time of the data to be written;
[0016] Accordingly, according to the current reference time corresponding to the time judgment condition and the data writing request, a data processing method for the data to be written is determined, including:
[0017] If the expected reading time is earlier than the current reference time, determining that the data processing mode of the data to be written is write processing;
[0018] If the expected reading time is later than the current reference time, it is determined that the data processing method for the data to be written is push processing.
[0019] In one embodiment, determining a data processing method for the data to be written according to the current reference time corresponding to the time judgment condition and the data writing request includes:
[0020] If the current time and the receiving time of the data writing request are both earlier than the current reference time, it is determined that the data processing mode of the data to be written is writing processing.
[0021] If the receiving time is earlier than the current reference time, and the current time is later than the current reference time, it is determined that the data processing method of the data to be written is push processing.
[0022] In one embodiment, the method further comprises:
[0023] In response to a data read request sent by the reading side, the current reference time is updated according to a preset time step, and the updated current reference time is used as the next reference time corresponding to the time judgment condition;
[0024] The data between the current reference time and the previous reference time is read, and the read data is sent to the reading side.
[0025] In one embodiment, determining a data processing method for data to be written according to a write judgment condition and a data write request includes:
[0026] In the case where the write determination condition is a number determination condition, a data processing method for the data to be written is determined according to the data number of the data to be written and the reference number corresponding to the number determination condition.
[0027] In a second aspect, the present application also provides a data processing device. The device includes:
[0028] A receiving module, used for receiving a data writing request sent by a writing side; wherein the data writing request carries the data to be written and data description information of the data to be written;
[0029] A first determination module, used to determine a write judgment condition of the data to be written according to the data description information of the data to be written;
[0030] A second determination module is used to determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0031] The processing module is used to process the data to be written according to the data processing method.
[0032] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0033] Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written;
[0034] Determine the writing judgment condition of the data to be written according to the data description information of the data to be written;
[0035] Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0036] The data to be written is processed according to the data processing method.
[0037] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0038] Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written;
[0039] Determine the writing judgment condition of the data to be written according to the data description information of the data to be written;
[0040] Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0041] The data to be written is processed according to the data processing method.
[0042] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0043] Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written;
[0044] Determine the writing judgment condition of the data to be written according to the data description information of the data to be written;
[0045] Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0046] The data to be written is processed according to the data processing method.
[0047] The above-mentioned data processing method, device, computer equipment and storage medium receive a data write request sent by the writing side that carries the data to be written and the data description information of the data to be written. According to the data description information of the data to be written, the write judgment condition of the data to be written is determined. According to the write judgment condition and the data write request, the data processing method of the data to be written is determined; wherein the data processing method includes write processing or push processing; according to the data processing method, the data to be written is processed. When processing the data to be written, the present application takes into account the data description information of the data to be written, and determines the write judgment condition of the data to be written according to the data description information. Based on the write judgment condition, the data processing method of the data to be written is determined, and the data processing method includes write processing or push processing, that is, when the data processing method does not meet the write processing condition, the data to be written can be directly pushed to the consumer side through push processing, which can effectively solve the problem that some of the data to be written cannot be read due to the existence of write delay when the consumer side reads the data. And compared with the traditional use of "distributed lock" to solve the problem of data not being able to read, the present application can support multiple terminal devices on the writing side to execute write tasks concurrently, effectively improving the data throughput. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] FIG1 is an application environment diagram of the data processing method provided by this embodiment;
[0049] Figure 2A schematic diagram of a flow chart of a first data processing method provided in this embodiment;
[0050] Figure 3 A schematic diagram of a flow chart of a first data processing method for determining data to be written provided in this embodiment;
[0051] Figure 4 A schematic diagram of a process of reading data by a reading side provided in this embodiment;
[0052] Figure 5 A schematic diagram of a flow chart of a second data processing method for determining data to be written provided in this embodiment;
[0053] Figure 6 A schematic diagram of a flow chart of a second data processing method provided in this embodiment;
[0054] Figure 7 A structural block diagram of a first data processing device provided in this embodiment;
[0055] Figure 8 A structural block diagram of a second data processing device provided in this embodiment;
[0056] FIG. 9 is a diagram showing the internal structure of a computer device provided in this embodiment. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0058] The data processing method provided in the embodiment of the present application can be applied in the application environment as shown in Figure 1, and is particularly suitable for the case where the write side has multiple generators (i.e., multiple terminal devices that can write data) and the read side has one consumer (i.e., one terminal device that can read data). Among them, the distributed database 104 receives the data write request sent by the write side 102. Among them, the data write request carries the data to be written and the data description information of the data to be written. The distributed database 104 determines the write judgment condition of the data to be written according to the data description information of the data to be written, and determines the data processing method of the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing. The distributed database 104 processes the data to be written according to the data processing method. If the data processing method is push processing, there is no need to perform write processing on the data to be written, and it is directly pushed to the consumer side 106.
[0059] Among them, a distributed database refers to a database system that stores data in multiple physical locations or nodes, and these nodes are interconnected through a network and work together to provide unified data access and management services.
[0060] The write side, also called the production side, refers to the party that writes data to the distributed database. In a distributed database, the write side has multiple terminal devices with data writing functions. The terminal devices can be mobile phones, computers, etc.
[0061] The reading side, also known as the consumption side, refers to the party that reads data from the distributed database. This application is mainly used in scenarios where there is only one terminal device on the reading side, that is, multiple producers but consumers. The terminal device on the reading side can be a mobile phone, computer, etc.
[0062] In one embodiment, a data processing method is provided, and the method is applied to the distributed database in FIG. 1 as an example for explanation. Figure 2 As shown, the following steps are included:
[0063] S201, receiving a data writing request sent by a writing side, wherein the data writing request carries data to be written and data description information of the data to be written.
[0064] Among them, a data write request refers to a request instruction sent to a distributed database when the write side has a data write demand. Data to be written refers to data that needs to be written to a distributed database. In a distributed database, written data is generally stored in the form of a table. Therefore, data written to a distributed database is also called row data. Data description information refers to information used to characterize data attributes and properties. Data description information can be a data type, a data identifier, or other data information used to distinguish different data.
[0065] An optional implementation of this embodiment is: the writing side generates a data writing request according to the data to be written and the data description information of the data to be written, and sends the data writing request to the distributed database.
[0066] Another optional implementation of this embodiment is: the terminal device on the writing side calls the client application corresponding to the distributed database, and sends a data writing request to the distributed database based on the client application.
[0067] S202, determining a write judgment condition for the data to be written according to the data description information of the data to be written.
[0068] The write judgment condition refers to a judgment condition used to judge the data processing method of the data to be written.
[0069] An optional implementation of this embodiment is: determine the data type of the data to be written according to the data description information of the data to be written, and determine the write judgment condition of the data to be written according to the data type. For example, if the data type is time data, the write judgment condition is a time judgment condition.
[0070] Another optional implementation of this embodiment is: determining the data source of the data to be written according to the data description information of the data to be written, and determining the writing judgment condition of the data to be written according to the data source.
[0071] Another optional implementation of this embodiment is: if the data description information indicates that the data to be written contains sequence data, then the write judgment condition of the data to be written is determined to be a number judgment condition. Exemplarily, based on the data description information, it is determined that the data to be written is numbered data (that is, the data to be written is numbered or contains a number), then the write judgment condition of the data to be written is determined to be a number judgment condition. In other words, if it can be determined based on the data description information that the data to be written contains sequence data with increasing or decreasing numbers, it can be determined that the write judgment condition of the data to be written is a number judgment condition. If necessary, the write judgment condition of such data can also be configured as a time judgment condition.
[0072] Another optional implementation of this embodiment is: if the data description information indicates that the data to be written does not contain sequence data, the writing judgment condition of the data to be written is determined to be a time judgment condition. Exemplarily, according to the data description information, it is determined that the data to be written is non-numbered data. Since such data cannot be judged based on the number whether the written data meets the requirements, the writing judgment condition of the data to be written is determined to be a time judgment condition.
[0073] S203, determining a data processing method for the data to be written according to the write judgment condition and the data write request, wherein the data processing method includes write processing or push processing.
[0074] The write process refers to the write process of the data to be written. The push process refers to the direct push of the data to be written to the reading side, that is, the consumption side.
[0075] Optionally, in this embodiment, based on the write judgment condition, it can be judged whether the data to be written meets the requirements, if it meets the requirements, the data processing mode of the data to be written is determined to be write processing. If it does not meet the requirements, the data processing mode of the data to be written is determined to be push processing.
[0076] S204: Process the data to be written according to the data processing method.
[0077] An optional implementation of this embodiment is: if the data processing mode is write processing, a write operation is performed on the data to be written, that is, the data to be written is written into a corresponding data row in the table storage system of the distributed database.
[0078] Another optional implementation of this embodiment is: if the data processing method is push processing, the data to be written is pushed to the reading side without performing the data writing operation.
[0079] The above data processing method receives a data write request sent by the writing side and carries the data to be written and the data description information of the data to be written. According to the data description information of the data to be written, the writing judgment condition of the data to be written is determined. According to the writing judgment condition and the data writing request, the data processing method of the data to be written is determined; wherein the data processing method includes writing processing or pushing processing; according to the data processing method, the data to be written is processed. When processing the data to be written, the present application takes into account the data description information of the data to be written, and determines the writing judgment condition of the data to be written according to the data description information. Based on the writing judgment condition, the data processing method of the data to be written is determined, and the data processing method includes writing processing or pushing processing, that is, when the data processing method does not meet the writing processing condition, the data to be written can be pushed directly to the consumer side through push processing, which can effectively solve the problem that some of the data to be written cannot be read due to the existence of writing delay when the consumer side reads the data. And compared with the traditional use of "distributed lock" to solve the problem of data not being able to read, the present application can support multiple terminal devices on the writing side to execute writing tasks concurrently, effectively improving the data throughput.
[0080] In one embodiment, when the writing judgment condition is a time judgment condition, in order to determine the data processing method for the data to be written, such as Figure 3 As shown, an optional implementation of S203 includes:
[0081] S301, when the write judgment condition is a time judgment condition, determine a data processing method for the data to be written according to a current reference time corresponding to the time judgment condition and a data write request.
[0082] The current reference time refers to a pre-configured time used as a basis for time judgment, and the current reference time is also the time when the reading side concentrates on reading the written data.
[0083] An optional implementation of this embodiment is: in this embodiment, the data write request also carries the expected reading time of the data to be written. On this basis, according to the current reference time corresponding to the time judgment condition and the data write request, an optional implementation of determining the data processing method of the data to be written is: if the expected reading time is earlier than the current reference time, the data processing method of the data to be written is determined to be write processing. If the expected reading time is later than the current reference time, the data processing method of the data to be written is determined to be push processing. Among them, the expected reading time is the time configured by the write side when the data to be written is expected to be read by the consumer side, and also refers to the time when the data to be written is expected to be written. Exemplarily, if the expected reading time is 14:30:24, and the current reference time is 14:30:25, because when the write operation is performed on the data to be written, the read side can read the data to be written when reading the data at the current reference time, therefore, the data processing method of the data to be written is determined to be write processing. If the expected reading time is 14:30:26, and the current reference time is 14:30:25, because if a write operation is performed on the data to be written, the reading side cannot read the data to be written when reading the data at the current reference time, which will cause similar data loss problems. Therefore, it is determined that the data processing method for the data to be written is push processing. Instead of performing a write operation on the data to be written, it is directly pushed to the consumer side, which effectively solves the problem of data loss.
[0084] Another optional implementation in this embodiment is: if the current moment and the reception time of the data write request are both earlier than the current reference moment, the data processing method of the data to be written is determined to be write processing. If the reception time is earlier than the current reference moment, and the current moment is later than the current reference moment, the data processing method of the data to be written is determined to be push processing. Among them, the current moment refers to the actual processing moment of the data to be written after the distributed database receives the data write request through the interface. Exemplarily, if the reception time is 14:30:20, the current moment is 14:30:18, and the current reference moment is 14:30:22, because, at the current moment, if a write operation is performed on the data to be written, the reading side can read the data to be written when reading the data at the current reference moment, therefore, the data processing method of the data to be written is determined to be write processing. If the receiving time is 14:30:23, the current time is 14:30:26, and the current reference time is 14:30:25, because if a write operation is performed on the data to be written at the current time, the reading side cannot read the data to be written when reading the data at the current reference time. Therefore, the data processing method for the data to be written is push processing, and the data to be written is pushed to the consumer side.
[0085] In this embodiment, when the write judgment condition is a time judgment condition, the data processing method of the data to be written can be accurately determined according to the current reference time corresponding to the time judgment condition and the data write request. In this embodiment, the concept of the current reference time is introduced, and the current reference time is used as the "water level". For the data to be written that meets the requirements corresponding to the current reference time, the data processing method is write processing, and for the data to be written that does not meet the requirements corresponding to the current reference time, the data processing method is push processing, which ensures the atomicity of the entire process and can effectively solve the problem of "lost data" while satisfying the concurrent execution of write tasks.
[0086] In one embodiment, when a data read request is received from a reading side, Figure 4 As shown, an optional implementation of a data processing method includes:
[0087] S401, in response to a data read request sent by a reading side, updating a current reference time according to a preset time step, and using the updated current reference time as a next reference time corresponding to a time judgment condition.
[0088] The data read request refers to the data read instruction sent to the distributed database when the reading side has the need to read data. The preset time step refers to the preset time step for updating the current reference time, which can be 2 seconds, 5 seconds, etc. The next reference time refers to the time after the current reference time is updated according to the time step.
[0089] Optionally, in this embodiment, a preset time step is added to the current reference time to update the current reference time, and the updated current reference time is used as the next reference time corresponding to the time judgment condition. For example, if the current reference time is 15:13:20 and the preset time step is 2 seconds, the next reference time is 15:13:22.
[0090] S402, reading data between the current reference time and the previous reference time, and sending the read data to the reading side.
[0091] The previous reference time refers to the time before the current reference time is updated.
[0092] Optionally, in this embodiment, a previous reference moment is determined based on the current reference moment and a preset time step. The data between the current reference moment and the previous reference moment is read, and the read data is sent to the reading side. In this embodiment, an optional implementation method for determining the previous reference moment based on the current reference moment and the preset time step is: based on the current reference moment, subtract the preset time step as the first reference moment. In this embodiment, an optional way to read the data between the current reference moment and the previous reference moment is: read the data between the current reference moment and the previous reference moment from the table storage system, and send the read data to the reading side.
[0093] In this embodiment, when a data read request is received from the reading side, the current reference time is first updated according to the preset time step, and the updated current reference time is used as the next reference time corresponding to the time judgment condition. Then the data between the current reference time and the previous reference time is read, and the read data is sent to the reading side. That is to say, in this application, when a data read request is received, the current reference time is updated first, so that the next reference time can be used as a new reference time during the reading process to determine the data processing method of the data to be written in the data write request received during the reading process, thereby increasing the stability and accuracy of the entire data processing process.
[0094] In one embodiment, when the write judgment condition is a number judgment condition, such as Figure 5 As shown, an optional implementation of S203 includes:
[0095] S501, when the write judgment condition is a number judgment condition, determining a data processing method for the data to be written according to the data number of the data to be written and the reference number corresponding to the number judgment condition.
[0096] Optionally, in this embodiment, when writing the data to be written, it is written in the form of increasing numbers. On this basis, if the data number in the data to be written meets the conditional requirements of the reference number corresponding to the number judgment condition, then the data processing method of the data to be written is determined to be write processing. If the data number in the data to be written does not meet the conditional requirements of the reference number corresponding to the number judgment condition, then the data processing method of the data to be written is determined to be push processing. Exemplarily, if the data to be written is the number of a production product, the reference number is HD1100-HD1105, which is a range value. If the data number in the data to be written is HD1103, then the data processing method of the data to be written is determined to be write processing. If the data number of the data to be written is HD1009, then if the write processing continues, it cannot be read. Therefore, the data processing method in this case is push processing.
[0097] In this embodiment, when the write judgment condition is a number judgment condition, the data processing method of the data to be written can be accurately determined according to the data number of the data to be written and the reference number corresponding to the number judgment condition. Based on this embodiment, the number judgment condition can also be used as a write judgment condition, which can solve the problem of "losing data" when reading number-type data.
[0098] In one embodiment, if Figure 6 As shown, an optional implementation of a data processing method in this embodiment includes:
[0099] S601, receiving a data write request sent by a write side and carrying data to be written and data description information of the data to be written.
[0100] S602: If the data description information indicates that the data to be written includes sequence data, the writing judgment condition of the data to be written is determined to be a number judgment condition, and S606 is executed.
[0101] S603: If the data description information indicates that the data to be written does not contain sequence data, determine that the writing judgment condition of the data to be written is a time judgment condition, and execute S604.
[0102] S604: If the expected reading time carried in the data writing request is earlier than the current reference time corresponding to the time judgment condition, determine that the data processing method for the data to be written is write processing.
[0103] S605: If the expected reading time carried in the data writing request is later than the current reference time corresponding to the time judgment condition, determine that the data processing mode for the data to be written is push processing.
[0104] S606, when the write judgment condition is a number judgment condition, determine a data processing method for the data to be written according to the data number of the data to be written and the reference number corresponding to the number judgment condition.
[0105] S607, processing the data to be written according to the data processing method.
[0106] S608, in response to the data read request sent by the reading side, updating the current reference time according to the preset time step, and using the updated current reference time as the next reference time corresponding to the time judgment condition.
[0107] S609, reading data between the current reference time and the previous reference time, and sending the read data to the reading side.
[0108] In this embodiment, a data write request carrying data to be written and data description information of the data to be written is received from the write side. According to the data description information of the data to be written, the write judgment condition of the data to be written is determined. According to the write judgment condition and the data write request, the data processing method of the data to be written is determined; wherein the data processing method includes write processing or push processing; according to the data processing method, the data to be written is processed. When processing the data to be written, the present application takes into account the data description information of the data to be written, and determines the write judgment condition of the data to be written according to the data description information. Based on the write judgment condition, the data processing method of the data to be written is determined, and the data processing method includes write processing or push processing, that is, when the data processing method does not meet the write processing condition, the data to be written can be directly pushed to the consumer side through push processing, which can effectively solve the problem that some of the data to be written cannot be read due to the existence of write delay when the consumer side reads the data. And compared with the traditional use of "distributed lock" to solve the problem of data not being able to read, the present application can support multiple terminal devices on the write side to execute write tasks concurrently, which effectively improves the data throughput while solving the problem that the data cannot be read.
[0109] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the data processing device embodiment provided below can refer to the limitations on the data processing method above, and will not be repeated here.
[0110] In one embodiment, if Figure 7 As shown, a data processing device 1 is provided, the device comprising:
[0111] The receiving module 10 is used to receive a data writing request sent by the writing side; wherein the data writing request carries the data to be written and the data description information of the data to be written;
[0112] A first determination module 20, used to determine a write judgment condition of the data to be written according to the data description information of the data to be written;
[0113] The second determination module 30 is used to determine the data processing mode of the data to be written according to the write judgment condition and the data write request; wherein the data processing mode includes write processing or push processing;
[0114] The processing module 40 is used to process the data to be written according to the data processing method.
[0115] In one embodiment, the Figure 7The first determining module 20 is further specifically used for:
[0116] If the data description information indicates that the data to be written includes sequence data, determining that the writing judgment condition of the data to be written is a numbering judgment condition;
[0117] If the data description information indicates that the data to be written does not include sequence data, it is determined that the writing judgment condition of the data to be written is a time judgment condition.
[0118] In one embodiment, the Figure 7 The second determining module 30 is further specifically used for:
[0119] In the case where the write judgment condition is a time judgment condition, a data processing method for the data to be written is determined according to the current reference time corresponding to the time judgment condition and the data write request.
[0120] Based on the above embodiment, the data write request also carries the expected read time of the data to be written. Figure 7 The second determining module 30 is further specifically used for:
[0121] If the expected reading time is earlier than the current reference time, determining that the data processing mode of the data to be written is write processing;
[0122] If the expected reading time is later than the current reference time, it is determined that the data processing method for the data to be written is push processing.
[0123] Based on the above embodiments, Figure 7 The second determining module 30 is further specifically used for:
[0124] If the current time and the receiving time of the data writing request are both earlier than the current reference time, it is determined that the data processing mode of the data to be written is writing processing.
[0125] If the receiving time is earlier than the current reference time, and the current time is later than the current reference time, it is determined that the data processing method of the data to be written is push processing.
[0126] In one embodiment, if Figure 8 As shown, a data processing device 1 further includes:
[0127] An updating module, configured to respond to a data reading request sent by the reading side, update the current reference time according to a preset time step, and use the updated current reference time as the next reference time corresponding to the time judgment condition;
[0128] The reading module is used to read the data between the current reference time and the previous reference time, and send the read data to the reading side.
[0129] In one embodiment, the Figure 7 The first determining module 20 is further specifically used for:
[0130] In the case where the write determination condition is a number determination condition, a data processing method for the data to be written is determined according to the data number of the data to be written and the reference number corresponding to the number determination condition.
[0131] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0132] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data processing method is implemented.
[0133] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0134] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0135] Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written;
[0136] Determine the writing judgment condition of the data to be written according to the data description information of the data to be written;
[0137] Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0138] The data to be written is processed according to the data processing method.
[0139] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining the writing judgment condition of the data to be written according to the data description information of the data to be written, including:
[0140] If the data description information indicates that the data to be written includes sequence data, determining that the writing judgment condition of the data to be written is a numbering judgment condition;
[0141] If the data description information indicates that the data to be written does not include sequence data, it is determined that the writing judgment condition of the data to be written is a time judgment condition.
[0142] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining a data processing method for the data to be written according to the write judgment condition and the data write request, including:
[0143] In the case where the write judgment condition is a time judgment condition, a data processing method for the data to be written is determined according to the current reference time corresponding to the time judgment condition and the data write request.
[0144] In one embodiment, when the processor executes the computer program, the following steps are further implemented: the data write request also carries the expected read time of the data to be written;
[0145] Accordingly, according to the current reference time corresponding to the time judgment condition and the data writing request, a data processing method for the data to be written is determined, including:
[0146] If the expected reading time is earlier than the current reference time, determining that the data processing mode of the data to be written is write processing;
[0147] If the expected reading time is later than the current reference time, it is determined that the data processing method for the data to be written is push processing.
[0148] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining a data processing method for the data to be written according to the current reference time corresponding to the time judgment condition and the data writing request, including:
[0149] If the current time and the receiving time of the data writing request are both earlier than the current reference time, it is determined that the data processing mode of the data to be written is writing processing.
[0150] If the receiving time is earlier than the current reference time, and the current time is later than the current reference time, it is determined that the data processing method of the data to be written is push processing.
[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0152] In response to a data read request sent by the reading side, the current reference time is updated according to a preset time step, and the updated current reference time is used as the next reference time corresponding to the time judgment condition;
[0153] The data between the current reference time and the previous reference time is read, and the read data is sent to the reading side.
[0154] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining a data processing method for the data to be written according to the write judgment condition and the data write request, including:
[0155] In the case where the write determination condition is a number determination condition, a data processing method for the data to be written is determined according to the data number of the data to be written and the reference number corresponding to the number determination condition.
[0156] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0157] Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written;
[0158] Determine the writing judgment condition of the data to be written according to the data description information of the data to be written;
[0159] Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0160] The data to be written is processed according to the data processing method.
[0161] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining the writing judgment condition of the data to be written according to the data description information of the data to be written, including:
[0162] If the data description information indicates that the data to be written includes sequence data, determining that the writing judgment condition of the data to be written is a numbering judgment condition;
[0163] If the data description information indicates that the data to be written does not include sequence data, it is determined that the writing judgment condition of the data to be written is a time judgment condition.
[0164] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a data processing method for the data to be written according to the write judgment condition and the data write request, including:
[0165] In the case where the write judgment condition is a time judgment condition, a data processing method for the data to be written is determined according to the current reference time corresponding to the time judgment condition and the data write request.
[0166] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the data write request also carries the expected read time of the data to be written;
[0167] Accordingly, according to the current reference time corresponding to the time judgment condition and the data writing request, a data processing method for the data to be written is determined, including:
[0168] If the expected reading time is earlier than the current reference time, determining that the data processing mode of the data to be written is write processing;
[0169] If the expected reading time is later than the current reference time, it is determined that the data processing method for the data to be written is push processing.
[0170] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a data processing method for the data to be written according to the current reference time corresponding to the time judgment condition and the data writing request, including:
[0171] If the current time and the receiving time of the data writing request are both earlier than the current reference time, it is determined that the data processing mode of the data to be written is writing processing.
[0172] If the receiving time is earlier than the current reference time, and the current time is later than the current reference time, it is determined that the data processing method of the data to be written is push processing.
[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0174] In response to a data read request sent by the reading side, the current reference time is updated according to a preset time step, and the updated current reference time is used as the next reference time corresponding to the time judgment condition;
[0175] The data between the current reference time and the previous reference time is read, and the read data is sent to the reading side.
[0176] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a data processing method for the data to be written according to the write judgment condition and the data write request, including:
[0177] In the case where the write determination condition is a number determination condition, a data processing method for the data to be written is determined according to the data number of the data to be written and the reference number corresponding to the number determination condition.
[0178] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0179] Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written;
[0180] Determine the writing judgment condition of the data to be written according to the data description information of the data to be written;
[0181] Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing;
[0182] The data to be written is processed according to the data processing method.
[0183] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining the writing judgment condition of the data to be written according to the data description information of the data to be written, including:
[0184] If the data description information indicates that the data to be written includes sequence data, determining that the writing judgment condition of the data to be written is a numbering judgment condition;
[0185] If the data description information indicates that the data to be written does not include sequence data, it is determined that the writing judgment condition of the data to be written is a time judgment condition.
[0186] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a data processing method for the data to be written according to the write judgment condition and the data write request, including:
[0187] In the case where the write judgment condition is a time judgment condition, a data processing method for the data to be written is determined according to the current reference time corresponding to the time judgment condition and the data write request.
[0188] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the data write request also carries the expected read time of the data to be written;
[0189] Accordingly, according to the current reference time corresponding to the time judgment condition and the data writing request, a data processing method for the data to be written is determined, including:
[0190] If the expected reading time is earlier than the current reference time, determining that the data processing mode of the data to be written is write processing;
[0191] If the expected reading time is later than the current reference time, it is determined that the data processing method for the data to be written is push processing.
[0192] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a data processing method for the data to be written according to the current reference time corresponding to the time judgment condition and the data writing request, including:
[0193] If the current time and the receiving time of the data writing request are both earlier than the current reference time, it is determined that the data processing mode of the data to be written is writing processing.
[0194] If the receiving time is earlier than the current reference time, and the current time is later than the current reference time, it is determined that the data processing method of the data to be written is push processing.
[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0196] In response to a data read request sent by the reading side, the current reference time is updated according to a preset time step, and the updated current reference time is used as the next reference time corresponding to the time judgment condition;
[0197] The data between the current reference time and the previous reference time is read, and the read data is sent to the reading side.
[0198] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a data processing method for the data to be written according to the write judgment condition and the data write request, including:
[0199] In the case where the write determination condition is a number determination condition, a data processing method for the data to be written is determined according to the data number of the data to be written and the reference number corresponding to the number determination condition.
[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0201] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data processing method, characterized in that: The method comprises: Receive a data write request sent by the write side; wherein the data write request carries the data to be written and data description information of the data to be written; Determining a write judgment condition for the data to be written according to the data description information of the data to be written; Determine a data processing method for the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing; The data to be written is processed according to the data processing method.
2. The method according to claim 1, characterized in that The step of determining a write judgment condition for the data to be written according to the data description information of the data to be written comprises: If the data description information indicates that the data to be written includes sequence data, determining that the writing judgment condition of the data to be written is a numbering judgment condition; If the data description information indicates that the data to be written does not include sequence data, it is determined that the writing judgment condition of the data to be written is a time judgment condition.
3. The method according to claim 1 or 2, characterized in that: The step of determining a data processing method for the data to be written according to the write judgment condition and the data write request includes: In the case where the write judgment condition is a time judgment condition, the data processing method of the data to be written is determined according to the current reference time corresponding to the time judgment condition and the data write request.
4. The method according to claim 3, characterized in that The data writing request also carries the expected reading time of the data to be written; Correspondingly, determining the data processing method of the data to be written according to the current reference time corresponding to the time judgment condition and the data writing request includes: If the expected reading time is earlier than the current reference time, determining that the data processing mode of the data to be written is write processing; If the expected reading time is later than the current reference time, it is determined that the data processing mode of the data to be written is push processing.
5. The method according to claim 3, characterized in that: The step of determining a data processing method for the data to be written according to the current reference time corresponding to the time judgment condition and the data writing request includes: If the current time and the receiving time of the data writing request are both earlier than the current reference time, determining that the data processing mode of the data to be written is writing processing; If the receiving time is earlier than the current reference time, and the current time is later than the current reference time, it is determined that the data processing mode of the data to be written is push processing.
6. The method according to claim 3, characterized in that The method further comprises: In response to a data read request sent by the reading side, the current reference time is updated according to a preset time step, and the updated current reference time is used as the next reference time corresponding to the time judgment condition; The data between the current reference time and the previous reference time is read, and the read data is sent to the reading side.
7. The method according to claim 2, characterized in that The step of determining a data processing method for the data to be written according to the write judgment condition and the data write request includes: In the case where the write judgment condition is a number judgment condition, a data processing method of the data to be written is determined according to the data number of the data to be written and the reference number corresponding to the number judgment condition.
8. A data processing device, characterized in that: include: A receiving module, configured to receive a data writing request sent by a writing side; wherein the data writing request carries the data to be written and data description information of the data to be written; A first determining module, used to determine a writing judgment condition of the data to be written according to the data description information of the data to be written; A second determination module is used to determine a data processing method of the data to be written according to the write judgment condition and the data write request; wherein the data processing method includes write processing or push processing; A processing module is used to process the data to be written according to the data processing method.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the data processing method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.