A method and device for data storage based on streaming computing

Abstract

The present invention discloses a method and device for data storage based on streaming computing, and relates to the field of big data technology. A specific implementation of the method includes: in response to receiving a computing request for streaming data of a target business, obtaining historical result data corresponding to the target business from a first storage location; determining the current result data corresponding to the target business according to the streaming data and the historical result data, and storing the current result data in the first storage location; judging whether a data synchronization event is triggered; in the case of triggering a data synchronization event, synchronizing the current result data to a second storage location, so as to use the second storage location to respond to a user's data query request for the target business. This implementation can reduce the storage resources consumed by data storage in high-performance and large data volume scenarios, and improve the performance of data query.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method and device for data storage based on streaming computing. Background Art

[0002] The existing streaming computing process stores detailed data or aggregated real-time result data in the database. However, in high-performance scenarios, storing detailed data will lead to storage consumption, increase hardware resource costs, and large amounts of data transmission will also cause performance degradation; in addition, in the process of computing large amounts of data, storing aggregated real-time result data in memory will cause data reading and writing frequencies to be too high. In mainstream memory storage, write operations take longer than read operations, resulting in a large amount of resources being occupied by write operations, affecting the performance of reading data. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and device for data storage based on streaming computing, which can reduce the storage resources consumed by data storage and ensure the performance of data query in high-performance and large-data volume scenarios.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for storing data based on streaming computing is provided, comprising:

[0005] In response to receiving a computation request for streaming data of a target service, acquiring historical result data corresponding to the target service from a first storage location;

[0006] Determine the current result data corresponding to the target service according to the streaming data and the historical result data, and store the current result data in the first storage location;

[0007] Determine whether to trigger a data synchronization event;

[0008] In the case of triggering a data synchronization event, the current result data is synchronized to a second storage location, so as to use the second storage location to respond to a user's data query request for the target business.

[0009] Optionally, the data synchronization event includes:

[0010] The historical result data and the current result data are not within the same preset data range; or, it is determined that the time interval between the historical result data and the current result data is greater than a preset time length.

[0011] Optionally, before determining whether to trigger a data synchronization event, the method further includes: determining the preset data range or the preset duration according to the numerical value of the streaming data and / or the request frequency of the calculation request.

[0012] Optionally, before determining whether to trigger a data synchronization event, the method further includes: setting a step threshold of the preset data range.

[0013] Optionally, acquiring historical result data corresponding to the target business from the first storage location includes:

[0014] Determine a current data period corresponding to the calculation request, and obtain historical result data of the target business in the current data period from the first storage location.

[0015] Optionally, the method further comprises:

[0016] Traverse each historical result data stored in the historical data period in the first storage location; determine whether there is result data that has not been synchronized to the second storage location among the historical result data; if so, synchronize the result data that has not been synchronized to the second storage location to the second storage location.

[0017] Optionally, both the first storage location and the second storage location are Redis clusters.

[0018] According to another aspect of an embodiment of the present invention, a device for storing data based on streaming computing is provided, including:

[0019] an acquisition module, in response to receiving a computation request for streaming data of a target business, acquiring historical result data corresponding to the target business from a first storage location;

[0020] A determination module, which determines the current result data corresponding to the target business according to the streaming data and the historical result data, and stores the current result data in the first storage location;

[0021] The judgment module judges whether a data synchronization event is triggered; if the data synchronization event is triggered, the current result data is synchronized to a second storage location, so as to use the second storage location to respond to the user's data query request for the target business.

[0022] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0023] one or more processors;

[0024] a storage device for storing one or more programs,

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for data storage based on streaming computing provided by the present invention.

[0026] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for data storage based on streaming computing provided by the present invention is implemented.

[0027] An embodiment of the above invention has the following advantages or beneficial effects: when receiving a calculation request for streaming data of a target business, the historical result data of the target business is obtained from the first storage location, and then the current result data is calculated to determine whether a data synchronization event is triggered. When the data synchronization event is triggered, the current result data is synchronized to the second storage location, so that the user can query the data corresponding to the target business at the second storage location. The method of the embodiment of the present invention sets a data synchronization event during the streaming calculation process, and when the data synchronization event is triggered, the real-time aggregated result data is synchronized from the first storage location to the second storage location, which saves storage, realizes the isolation of data storage and data query functions, and reduces the impact of write operations on read operations, thereby improving the performance of querying data and meeting high performance requirements.

[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0030] Figure 1 is a schematic diagram of the main process of a data storage method based on streaming computing according to an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of the main process of another method for storing data based on streaming computing according to an embodiment of the present invention;

[0032] Figure 3 It is a structural schematic diagram of a data storage based on stream computing according to an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram of a process of data storage and data query based on stream computing in an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of main modules of a data storage device based on streaming computing according to an embodiment of the present invention;

[0035] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0036] Figure 7It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] Figure 1 is a schematic diagram of the main process of the method for storing data based on streaming computing according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0039] Step S101: in response to receiving a calculation request for streaming data of a target service, obtaining historical result data corresponding to the target service from a first storage location;

[0040] Step S102: determining current result data corresponding to the target service according to the streaming data and the historical result data, and storing the current result data in a first storage location;

[0041] Step S103: Determine whether a data synchronization event is triggered; if so, execute step S104; if not, execute step S105.

[0042] Step S104: synchronizing the current result data to the second storage location, so as to use the second storage location to respond to the user's data query request for the target business;

[0043] Step S105: Do not synchronize the current result data to the second storage location.

[0044] The method in the embodiment of the present invention can be applied to high-performance, large data volume scenarios. Streaming computing can be cumulative computing under large data streams, and streaming data can be cumulative values, such as the cumulative result of the user's request amount within 24 hours, then each request amount is streaming data, that is, streaming data can be an incremental value, when a calculation request for streaming data is received, historical result data is obtained, and current result data, that is, cumulative results, are calculated or aggregated, and historical result data and streaming data can be added to obtain current result data, and current result data is stored in a first storage location, and the first storage location stores the result data of the streaming computing process, and when a data synchronization event is triggered, the current result data that meets the conditions is synchronized to the second storage location.

[0045] In an embodiment of the present invention, a data synchronization event may be when the historical result data and the current result data are not within the same preset data range. That is, when the historical result data and the current result data are not within the same preset data range, a data synchronization event will be triggered; if they are within the same preset range, the data synchronization event will not be triggered.

[0046] In an embodiment of the present invention, before determining whether to trigger a data synchronization event, multiple preset data ranges are set, and the multiple preset data ranges can be multiple continuous data ranges or multiple discontinuous data ranges. There is no overlapping data in the multiple preset data ranges. For example, the multiple preset data ranges can be multiple continuous data ranges such as 0-10, 10-15, 15-25, 25-50, or multiple discontinuous data ranges such as 0-10, 15-25, 30-50, wherein each preset data range includes the data at the right endpoint (i.e., the step threshold) but does not include the data at the left endpoint. Optionally, the preset data range can be set according to the numerical value of the streaming data. If the numerical value of the streaming data is between 5-10, the preset data range can be set to 0-10, 10-15, 15-25, etc.; if the numerical value of the streaming data is between 100-150, the preset data range can be set to 0-150, 150-300, 300-500, etc.

[0047] Optionally, before determining whether to trigger a data synchronization event, a step threshold of a preset data range is set, that is, multiple data ranges are in a step shape. If the historical result data and the current result data are not in the same preset data range, it means that the current result data and the historical result data are not on the same step, that is, the current result data exceeds or crosses the step threshold of the preset data range where the historical result data is located, and a data synchronization event will be triggered at this time.

[0048] Optionally, the step threshold can also be set according to the needs of the target business. For example, the calculation task is "calculate the user's request amount within 24 hours". When the target business is in use, it is configured to return true when the user's request amount within 24 hours is greater than 10. Then the step threshold can be set to 10 for subsequent business operations; when the user's first request amount within 24 hours is 5, 5 is stored in the first storage location; when the user's second request amount is 10, 5+10=15 is stored in the first storage location. Since 15 is greater than the step threshold of 10, a data synchronization event is triggered. When performing data synchronization, data synchronization is triggered every time the cumulative request amount reaches the step threshold, thereby reducing the frequency of data synchronization, improving the performance of data queries, and ensuring the data accuracy of the target business.

[0049] In an embodiment of the present invention, a data synchronization event can be determined when the time interval between historical result data and current result data is greater than a preset time length. That is, when the time interval between the time of determining the historical result data and the time interval between determining the current result data exceeds the preset time length, a data synchronization event will be triggered. If it does not exceed the preset time length, the data synchronization time will not be triggered. In other words, the data synchronization event is a timed trigger, and a data synchronization event is triggered once every preset time length. For example, if a data synchronization event is triggered once every 2 hours, the data synchronization event will not be triggered within the 2 hours of the interval.

[0050] Optionally, before determining whether to trigger a data synchronization event, a preset duration is determined based on the request frequency of the calculation request. If the request frequency of the calculation request is high, a shorter preset duration can be set. For example, if the frequency of the calculation request is once every 2-3 hours, the preset duration can be set to 6 hours; if the request frequency of the calculation request is low, a longer preset duration can be set. If the frequency of the calculation request is once every 10-12 hours, the preset duration can be set to 30 hours. By setting a preset duration based on the request frequency of the calculation request and triggering a data synchronization event when the preset duration is exceeded, the data in the second storage location can be made less than the data stored in the first storage location, thereby reducing the frequency of data synchronization and reducing the impact of data synchronization on data query, thereby improving the performance of data query.

[0051] In an embodiment of the present invention, obtaining historical result data corresponding to the target business from the first storage location includes: determining a current data period corresponding to the calculation request, and obtaining historical result data of the target business in the current data period from the first storage location.

[0052] In an embodiment of the present invention, the historical result data may be the latest historical result data of the target business. The current data period may be set according to business requirements, and the current data period may be a preset time range. By determining the preset time range corresponding to the calculation request, the historical result data of the target business within the preset time range may be obtained from the first storage location. Optionally, obtaining the historical result data corresponding to the target business from the first storage location may also include: obtaining the historical result data of the target business closest to the current moment from the first storage location.

[0053] In the embodiment of the present invention, Figure 2 As shown, the data storage method based on streaming computing also includes:

[0054] Step S201: traverse each historical result data stored in the historical data period in the first storage location;

[0055] Step S202: determine whether there is any result data in each historical result data that has not been synchronized to the second storage location; if so, execute step S203; if not, end the process.

[0056] Step S203: Synchronize the result data that has not been synchronized to the second storage location to the second storage location.

[0057] Since the data in the second storage location is part of the result data obtained by real-time aggregation or calculation in the first storage location, when the data is queried through the second storage location, there may be a situation where the query results cannot be obtained. In this case, in order to meet the query requirements, the historical result data in the first storage location that is not synchronized to the second storage location can be synchronized to the second storage location. Optionally, synchronization can be performed according to the historical data cycle. For example, at 24 o'clock every day, the various historical result data of the target business between 0 o'clock and 24 o'clock can be obtained, and then the result data in each historical result data that is not synchronized to the second storage location is synchronized to the second storage location to meet the user's data query needs in the second storage location.

[0058] In the embodiment of the present invention, the first storage location and the second storage location are both Redis clusters, and may also be other databases, such as MySQL.

[0059] The first storage location and the second storage location correspond to different Redis clusters, respectively, so that the first storage location and the second storage location are isolated, so that the read data operation and the write data operation correspond to different storage locations. Among them, the first storage location is mainly used for computing data storage, which can be called computing storage, and the second storage location is mainly used for data query, which can be called query storage. Through the dual storage method, it is possible to avoid the high-frequency write operations of computing and the read operations competing for the CPU, network and storage resources of storage resources, affecting the query performance.

[0060] In an embodiment of the present invention, the first storage location is used to store result data obtained by real-time aggregation or calculation during streaming computing, which saves storage resources, reduces costs, and can effectively avoid the defect of performance degradation caused by large-scale data transmission; the second storage location is used to store part of the result data synchronized from the first storage location, which saves storage resources, reduces the amount of data stored in the second storage location, reduces the impact of data synchronization on queries, improves data query performance, and can also synchronize other result data when there are other query requirements.

[0061] like Figure 3The figure shows a structural diagram of a data storage based on streaming computing in an embodiment of the present invention. Calculate the amount requested by the user within 24 hours, set multiple preset ranges of 0-10, 10-15, 15-25, 25-35, and the corresponding step thresholds are 10, 15, 25 and 35 respectively; set the data synchronization event to the current result data and the historical result data are not in the same preset range; trigger writing when receiving each request amount, and then write the accumulated calculation result to the first storage location; the first request amount is 5, write 5 to the first storage location; the second request amount is 15, write the accumulated calculation result 5+15=20 Write to the first storage location; determine that 20 and 5 are not on the same ladder, 20 crosses the ladder threshold 15, and trigger a data synchronization event at this time, then synchronize 20 to the second storage location, so that the result can be queried from the second storage location; the third request amount is 30, and the cumulative calculation result 30+20=50 is written to the first storage location, determine that 50 and 20 are not on the same ladder, 50 crosses the ladder threshold 35, and trigger a data synchronization event at this time, then synchronize 50 to the second storage location, so that the result can be queried from the second storage location.

[0062] like Figure 4 FIG. 1 is a flow chart of data storage and data query based on stream computing according to an embodiment of the present invention.

[0063] Data storage process based on streaming computing: in response to receiving a computing request for streaming data of a target business, obtaining historical result data corresponding to the target business from a first storage location; obtaining current result data according to the historical result data and the streaming data, and storing the current result data in the first storage location; determining whether the current result data and the historical result data are not within the same preset data range, if so, synchronizing the current result data to the second storage location; if not, terminating the process;

[0064] Data query process based on streaming computing: in response to receiving a data query request for a target business, generate a query identifier according to the data query request, query the corresponding query result from the second storage location according to the query identifier, obtain the task result corresponding to the data query request according to the query result, return the task result, and end the process.

[0065] The data storage method based on streaming computing provided by the embodiment of the present invention realizes data storage and query through dual storage, isolates the data storage of the first storage location from the data query of the second storage location, and synchronizes the data by triggering data synchronization, thereby saving storage, reducing the impact of data write operations on read operations, and improving the performance of data query, so as to retain relatively complete data for computing storage and ensure better performance for query storage, thereby meeting the data read and write operation performance in high-performance and large data volume scenarios.

[0066] like Figure 5 As shown, another aspect of the embodiment of the present invention provides a data storage device 500 based on streaming computing, including:

[0067] The acquisition module 501, in response to receiving a calculation request for streaming data of a target business, acquires historical result data corresponding to the target business from a first storage location;

[0068] A determination module 502 determines current result data corresponding to the target business according to the streaming data and the historical result data, and stores the current result data in a first storage location;

[0069] The judgment module 503 judges whether a data synchronization event is triggered; if so, the current result data is synchronized to the second storage location to use the second storage location to respond to the user's data query request for the target business; otherwise, the current result data is not synchronized to the second storage location.

[0070] In the embodiment of the present invention, the data synchronization event includes: the historical result data and the current result data are not within the same preset data range; or it is determined that the time interval between the historical result data and the current result data is greater than a preset time length.

[0071] In the embodiment of the present invention, the judgment module 503 is further used to: determine a preset data range or a preset duration according to the numerical value of the streaming data and / or the request frequency of the calculation request before judging whether to trigger a data synchronization event.

[0072] In the embodiment of the present invention, the determination module 503 is further configured to: before determining whether to trigger a data synchronization event, set a step threshold of a preset data range.

[0073] In the embodiment of the present invention, the acquisition module 501 is further used to: determine the current data period corresponding to the calculation request, and acquire the historical result data of the target business in the current data period from the first storage location.

[0074] In an embodiment of the present invention, the judgment module 503 is also used to: traverse each historical result data stored in the historical data period in the first storage location; determine whether there is result data in each historical result data that has not been synchronized to the second storage location; if so, synchronize the result data that has not been synchronized to the second storage location to the second storage location.

[0075] In the embodiment of the present invention, the first storage location and the second storage location are both Redis clusters.

[0076] Another aspect of an embodiment of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of data storage based on streaming computing of an embodiment of the present invention.

[0077] Another aspect of an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method for data storage based on streaming computing according to an embodiment of the present invention is implemented.

[0078] Figure 6 An exemplary system architecture 600 is shown to which the method for storing data based on streaming computing or the apparatus for storing data based on streaming computing according to an embodiment of the present invention can be applied.

[0079] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0080] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0081] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0082] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0083] It should be noted that the method for data storage based on streaming computing provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the device for data storage based on streaming computing is generally set in the server 605 .

[0084] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0085] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0086] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0087] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0088] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0089] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0090] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0091] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including an acquisition module, a determination module, and a judgment module. The names of these modules do not, in certain cases, constitute limitations on the modules themselves, for example, the acquisition module may also be described as "a module that, in response to receiving a computational request for streaming data for a target business, acquires historical result data corresponding to the target business from a first storage location".

[0092] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: in response to receiving a calculation request for streaming data of a target business, obtaining historical result data corresponding to the target business from a first storage location; determining current result data corresponding to the target business based on the streaming data and the historical result data, and storing the current result data in a first storage location; determining whether a data synchronization event is triggered; and in the case of triggering a data synchronization event, synchronizing the current result data to a second storage location, so as to use the second storage location to respond to a user's data query request for the target business.

[0093] According to the technical solution of the embodiment of the present invention, data storage and query are realized through dual storage, the data storage of the first storage location is separated from the data query of the second storage location, and the data is synchronized by triggering data synchronization, which saves storage, reduces the impact of data write operations on read operations, improves the performance of data query, and achieves the retention of relatively complete data for computing storage, and ensures better performance for query storage, meeting the data read and write operation performance in high-performance and large data volume scenarios.

[0094] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data storage based on streaming computing, characterized in that: include: In response to receiving a computation request for streaming data of a target service, acquiring historical result data corresponding to the target service from a first storage location; Determine the current result data corresponding to the target service according to the streaming data and the historical result data, and store the current result data in the first storage location; Determine whether to trigger a data synchronization event; In the case of triggering a data synchronization event, synchronizing the current result data to a second storage location, so as to respond to a user's data query request for the target business using the second storage location; The data synchronization event includes: the historical result data and the current result data are not within the same preset data range; or it is determined that the time interval between the historical result data and the current result data is greater than a preset time length.

2. The method according to claim 1, characterized in that Before determining whether to trigger a data synchronization event, the method further includes: determining the preset data range or the preset duration according to the numerical value of the streaming data and / or the request frequency of the calculation request.

3. The method according to claim 1, characterized in that Before determining whether to trigger a data synchronization event, the method further includes: setting a step threshold of the preset data range.

4. The method according to claim 1, characterized in that: Acquiring historical result data corresponding to the target business from the first storage location includes: Determine a current data period corresponding to the calculation request, and obtain historical result data of the target business in the current data period from the first storage location.

5. The method according to claim 4, characterized in that The method further comprises: Traverse each historical result data stored in the historical data period in the first storage location; determine whether there is result data that has not been synchronized to the second storage location among the historical result data; if so, synchronize the result data that has not been synchronized to the second storage location to the second storage location.

6. The method according to any one of claims 1 to 5, characterized in that: The first storage location and the second storage location are both Redis clusters.

7. A data storage device based on streaming computing, characterized in that: include: an acquisition module, in response to receiving a computation request for streaming data of a target business, acquiring historical result data corresponding to the target business from a first storage location; A determination module, which determines the current result data corresponding to the target business according to the streaming data and the historical result data, and stores the current result data in the first storage location; A judgment module determines whether a data synchronization event is triggered; In the case of triggering a data synchronization event, synchronizing the current result data to a second storage location, so as to respond to a user's data query request for the target business using the second storage location; The data synchronization event includes: the historical result data and the current result data are not within the same preset data range; Alternatively, it is determined that the time interval between the historical result data and the current result data is greater than a preset time length.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images