Method and device for orderly providing data
By setting intermediate units cache data blocks and their sequence numbers in a distributed data system, the problem that data producers cannot guarantee global order is solved, and the orderly provision of data is achieved, saving data consumer resources and simplifying order is achieved. Sexual maintenance.
Patent Information
- Application Number
- CN202411028274.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
AI Technical Summary
In distributed data systems or multi-process systems, multiple data producers cannot guarantee the global order of data, resulting in the inability to apply to usage scenarios that require data order, such as model training tasks.
By setting an intermediate unit between the data producer and the data consumer, the data blocks and sequence numbers received from multiple data producers are cached, and the data blocks are provided to the data consumers in the order of sequence numbers, thereby ensuring the global order of the data. sex.
It realizes the global orderliness of data provision, saves the computing resources and storage resources required by data consumers, and simplifies the process of maintaining orderly data consumers.
Smart Images

Figure CN118964441A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present specification belong to the field of computer technology, and more particularly, to a method and device for providing data in an orderly manner. Background Art
[0002] In a distributed data system or a multi-process system, there are usually multiple data producers. The multiple data producers can generate multiple data in parallel and provide the data to the data consumers in parallel. However, when the multiple data producers provide data to the data consumers, the global order of the data cannot be guaranteed. Therefore, the above system cannot be applied to the use scenarios where the data needs to be input to the data consumers in an orderly manner (such as model training tasks that require data order). Summary of the invention
[0003] The object of the present invention is to provide a solution for providing data in an orderly manner.
[0004] A first aspect of the present specification provides a method for providing data in an orderly manner, which is performed by an intermediate unit, and the method comprises:
[0005] receiving a first data block and a first sequence number thereof from any data production unit among the plurality of data production units, wherein the first sequence number is one of a plurality of consecutive sequence numbers, and the plurality of consecutive sequence numbers respectively correspond to a plurality of data blocks generated by the plurality of data production units;
[0006] storing the first data block and its first sequence number in association with each other;
[0007] The first data block is provided to a data consumption unit according to the arrangement position of the first sequence number in the multiple consecutive sequence numbers.
[0008] In one embodiment, the intermediate unit is provided with a sliding window, a window size of the sliding window, and a current minimum window number of the sliding window, wherein the sliding window includes a sequence number of at least one data block currently stored in the intermediate unit.
[0009] The associative storage of the first data block and its first sequence number includes:
[0010] Determine, based on the window size and the minimum window number, a first sequence number range currently corresponding to the sliding window;
[0011] When it is determined that the first sequence number is within the first sequence number range and the sliding window does not include the first sequence number, storing the first data block and the first sequence number in an associated manner;
[0012] The first sequence number is added to the sliding window.
[0013] In one embodiment, the method further comprises:
[0014] receiving a third data block and a third sequence number thereof from a first data production unit among the plurality of data production units;
[0015] When it is determined that the third sequence number is after the second sequence number range currently corresponding to the sliding window, first feedback information is returned to the first data production unit, where the first feedback information is used to instruct the first data production unit to re-upload the third data block and its third sequence number.
[0016] In one embodiment, the minimum window number is the first sequence number, and the method further includes: after determining that the data consumption unit has obtained the first data block, deleting the first data block in the intermediate unit, deleting the first sequence number in the sliding window, and updating the minimum window number.
[0017] In one embodiment, the intermediate unit is connected to multiple data consumption units, and after determining that the data consumption unit has obtained the first data block, the intermediate unit deletes the first data block, deletes the first sequence number in the sliding window, and updates the minimum window number, including: after determining that the multiple data consumption units have all obtained the first data block, the intermediate unit deletes the first data block, deletes the first sequence number in the sliding window, and updates the minimum window number.
[0018] In one embodiment, the intermediate unit stores a consumption state corresponding to the data consumption unit, and the consumption state includes the first sequence number, which is used to indicate that the data consumption unit is about to consume the first data block.
[0019] Providing the first data block to a data consumption unit according to the arrangement position of the first sequence number in the plurality of consecutive sequence numbers comprises:
[0020] In response to a consumption request from the data consumption unit, determining whether the intermediate unit stores the first data block according to the consumption status;
[0021] In case it is determined that the intermediate unit stores the first data block, providing the first data block to the data consumption unit;
[0022] The consumption status is updated.
[0023] In one implementation, the determining that the data consumption unit has acquired the first data block includes, according to the consumption state, determining that the data consumption unit has acquired the first data block.
[0024] In one implementation, in the case where the consumption status includes only the first sequence number, updating the consumption status includes updating the consumption status to a sequence number after the first sequence number;
[0025] In the case where the consumption state includes a plurality of sequence numbers, updating the consumption state includes deleting the first sequence number in the consumption state.
[0026] In one implementation, the consumption status includes the first sequence number and the second sequence number, the first sequence number is smaller than the second sequence number, and the first sequence number and the second sequence number are consecutive sequence numbers.
[0027] Providing the first data block to a data consumption unit according to the arrangement position of the first sequence number in the plurality of consecutive sequence numbers comprises:
[0028] In response to a consumption request from the data consumption unit, determining whether the intermediate unit stores the first data block and the second data block corresponding to the second sequence number according to the consumption status;
[0029] When it is determined that the intermediate unit stores the second data block but does not store the first data block, providing the second data block to the data consumption unit;
[0030] The second sequence number is deleted in the consumption state.
[0031] In one embodiment, the first data block and its first sequence number are obtained by any data production unit through the following process: receiving initial data and its corresponding first sequence number from a data distribution unit, and obtaining the first data block and the first sequence number based on the initial data and the first sequence number.
[0032] A second aspect of the present specification provides a device for providing data in an orderly manner, comprising:
[0033] A receiving unit, configured to receive a first data block and a first sequence number thereof from any data production unit among the plurality of data production units, wherein the first sequence number is one of a plurality of consecutive sequence numbers, and the plurality of consecutive sequence numbers respectively correspond to a plurality of data blocks generated by the plurality of data production units;
[0034] A storage unit, used for storing the first data block and its first sequence number in an associated manner;
[0035] A providing unit is used to provide the first data block to a data consumption unit according to the arrangement position of the first sequence number in the multiple consecutive sequence numbers.
[0036] A third aspect of the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described in the first aspect.
[0037] A fourth aspect of the specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.
[0038] The fifth aspect of this specification provides a computer program product, including a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.
[0039] The solution in the embodiments of this specification sets an intermediate unit between multiple data production units and data consumption units, and the intermediate unit caches data blocks and their sequence numbers received from multiple data production units, so that data blocks can be provided to the data consumption units in the order of sequence numbers, thereby ensuring the global orderliness of data provision. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 A system architecture diagram in an embodiment of this specification;
[0042] Figure 2 A flow chart of a method for orderly providing data in an embodiment of this specification;
[0043] Figure 3 This is a schematic diagram of a process of flow control by an intermediate unit in an embodiment of this specification;
[0044] Figure 4 The present invention is a structural diagram of a device for providing data in an orderly manner in one embodiment of the present specification. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.
[0046] In order to solve the problems existing in the related art as described above, the embodiments of this specification set an intermediate unit between multiple data production units and data consumption units, and the intermediate unit caches the data blocks and their sequence numbers received from the multiple data production units, so that the data blocks can be provided to the data consumption units in the order of the sequence numbers, thereby ensuring the global orderliness of data provision. Among them, the data block is a block of a preset size into which the data is divided during the storage or transmission process.
[0047] The data production unit and the data consumption unit are responsible for the production and consumption of data blocks respectively. The data production unit can be an application, service, device, thread or other source that can produce data blocks. The data blocks generated by the data production unit may include data read by sensors, information input by users, application logs, transaction records, etc. The data consumption unit can be an application, service, analysis tool, thread or other component that needs to obtain data blocks. The data consumption unit can process the received data blocks, including data parsing, filtering, conversion, storage, analysis and other operations.
[0048] Figure 1 is a system architecture diagram in the embodiments of this specification. Figure 1 As shown, the system may include: multiple data production units, Figure 1 Data production units 11, 12 and 13 are shown as examples; an intermediate unit 20; a plurality of data consumption units, Figure 1 The data consumption units 31, 32 and 33 are shown as examples. The data production unit may generate a data block and its corresponding sequence number, and send the data block and its sequence number to the intermediate unit 20. For example, the data production unit 11 generates a data block with a sequence number of 1 ( Figure 1The data block is generated by the intermediate unit 20 and sent to the intermediate unit 20 together with its sequence number. The data production unit 12 generates a data block with a sequence number of 4 and sends the data block with its sequence number to the intermediate unit 20. The data production unit generates a data block with a sequence number of 6 and sends the data block with its sequence number to the intermediate unit 20. The intermediate unit 20 can be an entity such as an application, a service, a device, or a thread. For example, the intermediate unit 20 can be an intermediate device between nodes in a distributed data system, or it can be an intermediate thread between threads in a multi-process system. The intermediate unit 20 includes a cache space for storing data blocks and their sequence numbers received from each data production unit. Any data consumption unit can request a data block from the intermediate unit 20, so that the intermediate unit 20 can provide the data block to the data consumption unit in the order of the sequence numbers of the data blocks stored therein. Among them, the above-mentioned multiple data blocks can have the same size, or can also have different sizes.
[0049] In one embodiment, the data production units 11, 12 and 13 can be connected to the same front-end unit, which can obtain the initial data and its sequence number. For example, the front-end unit is a shopping platform, and each initial data is the transaction data of the shopping platform in each time period. The sequence number of each transaction data is determined based on the order of the time periods corresponding to the transaction data. Therefore, the sequence numbers of each initial data are arranged from small to large and will not be repeated. For example, the front-end unit can set the sequence number of the transaction data of the first time period to 1, and incrementally set the sequence number of the transaction data of the second time period to 2, and so on, until the sequence number of the transaction data of the nth time period is set to n. Afterwards, the front-end unit can distribute each initial data and its sequence number to each data production unit according to a preset rule while obtaining each initial data. After receiving the initial data and its sequence number, each data production unit can perform preset processing on the initial data, such as abnormal data deletion, to obtain a data block to be sent to the data consumption unit. In this scenario, each data consumption unit can be, for example, a neural network model, and each data block needs to be input into any neural network model in the order of the serial numbers of each data block, so as to predict a time-related characteristic of the shopping platform, such as the sales of the shopping platform in the next month, the promotion strategy of the shopping platform in the next quarter, etc.
[0050] In another embodiment, each data production unit processes part of the initial data in the big data according to a preset rule, and the data production unit can determine the serial number corresponding to the part of the initial data according to the position of the part of the initial data processed by it in the big data, and can process the part of the initial data to obtain the data block to be provided to the data consumption unit. For example, the big data can be divided into N initial data with 10M as the data volume unit, and each data production unit can determine which data of the N initial data it will obtain by taking N modulo 3, wherein the arrangement order of the above N initial data in the big data can be used as the serial number of the initial data. For example, the serial number of the first 10M of initial data in the big data is 1, and the serial number of the second 10M of initial data in the big data is 2, and so on. Wherein the big data is, for example, a long text, and each data consumption unit is, for example, each text processing model, and each data block needs to be input into any text processing model according to the order of the position in the long text corresponding to each data block, that is, each data block needs to be input into any text processing model according to the order of the serial number of each data block from small to large.
[0051] According to the above two examples, it can be seen that the sequence numbers of the multiple data blocks generated by the multiple data production units are usually multiple sequence numbers that increase continuously. It can be understood that the embodiments of this specification are not limited to this. For example, the sequence numbers of the multiple data blocks can also be multiple sequence numbers that decrease continuously.
[0052] Figure 2 This is a flow chart of a method for orderly providing data in an embodiment of this specification. The method can be performed by Figure 1 The intermediate unit in is executed.
[0053] like Figure 2 As shown, first, in step S201, the intermediate unit receives data blocks and their sequence numbers from multiple data production units, and in step S203, stores the data blocks and their sequence numbers.
[0054] like Figure 1 As shown, the intermediate unit can receive different data blocks and their sequence numbers from, for example, data production units 11, 12, and 13 in parallel. Wherein, as described above, the different data blocks have non-repetitive sequence numbers, that is, the sequence number is a global sequence number relative to multiple data production units, the sequence number is one of multiple continuous sequence numbers, and the multiple continuous sequence numbers correspond to multiple data blocks that have been generated by multiple data production units. Afterwards, the intermediate unit can associate the different data blocks and their corresponding sequence numbers and store them in a cache space for providing to the data consumption unit. Wherein, when setting the cache space, the intermediate unit 20 can specify the capacity of the cache space.
[0055] In one embodiment, since the capacity of the cache space of the intermediate unit 20 is limited, it is necessary to control the flow of data blocks received from each data production unit. The intermediate unit 20 can perform flow control through a sliding window data structure. Figure 3 The process of flow control performed by the intermediate unit 20 is described.
[0056] like Figure 3 As shown, a sliding window w1 and a window size corresponding to the sliding window w1 can be set in the intermediate unit 20. The window size is set based on the capacity of the cache space. For example, the size of the sliding window w1 can indicate the number of data blocks allowed to be received, for example, the size of the sliding window w1 is "6", which means that 6 data blocks are allowed to be received. Alternatively, the size of the sliding window w1 can indicate the amount of data allowed to be received, for example, the size of the sliding window w1 is 100 megabytes. The current minimum window number of the sliding window w1 is also maintained in the intermediate unit 20. Initially, a default value can be set for the minimum window number, such as "1", which means that the sequence number of the data block increases from 1. The sliding window w1, for example, includes a list of sequence numbers of the data blocks currently stored in the intermediate unit 20, wherein in the list, the sequence numbers are arranged in ascending order, for example, as Figure 3 As shown in , the sliding window w1 includes sequentially arranged sequence numbers ② and ③, which indicates that data block 2 and data block 3 are stored in the cache space.
[0057] The intermediate unit 20 can determine whether the data currently received from each data production unit can be stored in the cache space based on the window size and the minimum window number. Figure 3 , when the minimum window number is 1 and the window size is "6" (indicating 6 data blocks), the intermediate unit 20 can determine that data blocks 1 to 6 are currently allowed to be stored. In another embodiment, when the window size is a threshold data amount, in this case, assuming that the size of each data block is a fixed size, the intermediate unit can determine the range of data blocks currently allowed to be stored by the intermediate unit 20, such as data blocks 1 to 6, according to the minimum window number, the threshold data amount and the size of the data block. After receiving data block 1 from the data production unit 11, the intermediate unit determines that data block 1 is within the range of data blocks 1 to 6, and the sliding window w1 does not include the sequence number ① corresponding to data block 1, so that it can be determined that data block 1 can be stored in the cache space. After storing data block 1 in the cache space, the intermediate unit 20 can update the sliding window w1, that is, in the list of the sliding window w1, the sequence number corresponding to data block 1 is added to the list according to the size of the sequence number, specifically, the sequence number ① is added at the first position of the list to indicate that data block 1 is currently stored in the cache space.
[0058] After receiving data block 3 from data production unit 12, the intermediate unit determines that data block 3 is within the range of data block 1 to data block 6, but the sliding window w1 already includes sequence number ③, that is, data block 3 has been stored in the cache space. The intermediate unit 20 can determine that data block 3 is a repeatedly sent data block, and therefore can send feedback information to the data production unit 12, which indicates that data block 3 is a repeatedly sent data block.
[0059] After receiving data block 7 from data production unit 13, the intermediate unit 20 determines that data block 7 is within the range of data block 1 to data block 6. The intermediate unit 20 may send feedback information to the data production unit 13, and the feedback information is used to instruct the data production unit 13 to retransmit data block 7 after a preset time.
[0060] In step S205, the intermediate unit provides the data block to the data consumption unit according to the arrangement position of the sequence number of the data block in the plurality of continuous sequence numbers.
[0061] In one embodiment, each data consumption unit does not have the ability to cache data, and the intermediate unit 20 needs to provide data blocks to any data consumption unit in order from the smallest sequence number to the largest sequence number. The intermediate unit 20 can provide data blocks to the data consumption unit in response to the pull request of each data consumption unit for the data block. The intermediate unit 20 can record the consumption status corresponding to each data consumption unit, and the consumption status is used to indicate which data block the data consumption unit will consume.
[0062] For example, taking the data consumption unit 31 as an example, when the data consumption unit 31 has not started to consume data, the intermediate unit 20 uses the record "①" corresponding to the data consumption unit 31 as the consumption state, which is used to indicate that the data consumption unit 31 is about to consume data block 1. When the data consumption unit 31 first requests the intermediate unit 20 to pull the data block, the intermediate unit 20 can determine whether the sliding window w1 includes "①", that is, whether the data block 1 is stored in the cache space, according to the sequence number "①" in the consumption state corresponding to the data consumption unit 31. In the case of determining that the sliding window w1 includes "①", the data block 1 corresponding to the sequence number (i.e., sequence number 1) is provided from the cache space to the data consumption unit 31. In another case, when the intermediate unit 20 determines that the sliding window w1 does not include "①", the intermediate unit 20 can instruct the data consumption unit 31 to request the data block 1 again after a preset time. After providing data block 1 to data consumption unit 31 , the intermediate unit may update the consumption state of data consumption unit 31 to “②” to indicate that the next data block consumed by data consumption unit 31 is data block 2 .
[0063] After determining that the consumption status of data consumption units 31, 32 and 33 are all updated to "②", the intermediate unit 20 can determine that all data consumption units have consumed data block 1, so it can delete data block 1 in the cache space, delete ① in the list of the sliding window, and update the minimum window number of the sliding window w1 to "2". Through the above operations, the sliding window w1 is slid backward. For example, after the minimum window number of the sliding window w1 is updated to 2, the intermediate unit 20 can re-determine the range of data blocks currently allowed to be stored as data blocks 2 to data blocks 7 based on the minimum window number and the window size. Therefore, when the data production unit 13 sends data block 7 to the intermediate unit 20 again, the intermediate unit 20 can determine that data block 7 is currently allowed to be stored, so that data block 7 is stored in the cache space, and the sequence number ⑦ is increased in the list of the sliding window w1 accordingly.
[0064] It can be understood that the embodiments of the present specification are not limited to providing data to multiple data consumption units. In the case where the intermediate unit provides data to a data consumption unit (for example, data consumption unit 31), after the intermediate unit 20 determines that the data consumption unit 31 has consumed data block 1, it can slide the sliding window w1 backward by one sequence number as described above.
[0065] In another embodiment, each data consumption unit is allowed to cache a certain number of data blocks, for example, one data block can be cached. In this case, the intermediate unit 20 can set two sequence numbers in the consumption state of each data consumption unit. Figure 3 As shown, when the data consumption unit 31 has not started to consume data blocks, the sequence numbers ① and ② can be set in the consumption state s21 of the data consumption unit 31, indicating that data block 1 and / or data block 2 can be provided to the data consumption unit 31 at present. In this case, the intermediate unit 20, for example, can provide data block 2 to the data consumption unit 31 first when it receives data block 2 first and does not receive data block 1, without providing data block 2 after providing data block 1. After the intermediate unit 20 provides data block 2 to the data consumption unit 31 first, the sequence number ② in the consumption state s21 can be deleted. When the data consumption unit 31 receives data block 2 first and does not receive data block 1, the data consumption unit 31 can cache data block 2, and after receiving data block 1, sequentially process data block 1 and data block 2, for example, sequentially input data block 1 and data block 2 into the neural network model. After the intermediate unit provides both data block 1 and data block 2 indicated by the consumption state s21 to the data consumption unit 31, the consumption state s21 can be updated to sequence numbers ③ and ④.
[0066] In the case where the consumption state includes multiple sequence numbers, the intermediate unit 20 can slide the sliding window w1 in the same manner according to the records in the consumption state. For example, after the intermediate unit 20 determines that each data consumption unit has consumed data block 1 according to each consumption state, similar to the above description, the data block 1 is deleted from the cache space, and the sliding window w1 is moved back by one sequence number, and so on.
[0067] In the above-mentioned orderly data provision scheme, the point-to-point data transmission between the data producer and the data consumer is converted into data transmission at three ends: the data producer, the intermediate unit, and the data consumer, thereby using the intermediate unit to ensure the global orderliness of data transmission. In addition, through this scheme, data consumers do not need to maintain orderliness by themselves, but only need to continuously pull data blocks from the intermediate unit until receiving feedback that the data block consumption is completed, thus saving the computing resources and storage resources required by data consumers.
[0068] Figure 4 This is an architectural diagram of a device for providing data in an orderly manner in an embodiment of this specification, wherein the device is used to perform the following steps: Figure 2 The method shown comprises:
[0069] A receiving unit 41 is configured to receive a first data block and a first sequence number thereof from any data production unit among the plurality of data production units, wherein the first sequence number is one of a plurality of consecutive sequence numbers, and the plurality of consecutive sequence numbers respectively correspond to a plurality of data blocks generated by the plurality of data production units;
[0070] A storage unit 42, configured to store the first data block and its first sequence number in association with each other;
[0071] The providing unit 43 is used to provide the first data block to a data consumption unit according to the arrangement position of the first sequence number in the multiple consecutive sequence numbers.
[0072] The embodiment of the present specification also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the following Figure 2 The method shown.
[0073] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the following is implemented: Figure 2 The method shown.
[0074] The embodiments of the present specification also provide a computer program product, including a computer program / instruction, which is executed by a processor to implement the following Figure 2 The steps of the method shown.
[0075] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0076] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0077] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0078] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.
[0079] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0080] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0083] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0084] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0085] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0086] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0087] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0088] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.
[0089] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.
Claims
1. A method for providing data in an orderly manner, performed by an intermediate unit, the method comprising: receiving a first data block and a first sequence number thereof from any data production unit among the plurality of data production units, wherein the first sequence number is one of a plurality of consecutive sequence numbers, and the plurality of consecutive sequence numbers respectively correspond to a plurality of data blocks generated by the plurality of data production units; storing the first data block and its first sequence number in association with each other; The first data block is provided to a data consumption unit according to the arrangement position of the first sequence number in the multiple consecutive sequence numbers.
2. The method according to claim 1, wherein the intermediate unit is provided with a sliding window, a window size of the sliding window, and a current minimum window number of the sliding window, wherein the sliding window includes a sequence number of at least one data block currently stored in the intermediate unit, The associative storage of the first data block and its first sequence number includes: Determine, based on the window size and the minimum window number, a first sequence number range currently corresponding to the sliding window; When it is determined that the first sequence number is within the first sequence number range and the sliding window does not include the first sequence number, storing the first data block and the first sequence number in an associated manner; The first sequence number is added to the sliding window.
3. The method according to claim 2, further comprising: receiving a third data block and a third sequence number thereof from a first data production unit among the plurality of data production units; When it is determined that the third sequence number is after the second sequence number range currently corresponding to the sliding window, first feedback information is returned to the first data production unit, where the first feedback information is used to instruct the first data production unit to re-upload the third data block and its third sequence number.
4. The method according to claim 2, wherein the minimum window number is the first sequence number, and the method further comprises: After determining that the data consumption unit has acquired the first data block, the first data block is deleted in the intermediate unit, the first sequence number is deleted in the sliding window, and the minimum window number is updated.
5. The method according to claim 4, wherein the intermediate unit is connected to a plurality of the data consumption units, and after determining that the data consumption unit has acquired the first data block, the intermediate unit deletes the first data block, deletes the first sequence number in the sliding window, and updates the minimum window number, comprising: After determining that the plurality of data consumption units have acquired the first data block, the first data block is deleted in the intermediate unit, the first sequence number is deleted in the sliding window, and the minimum window number is updated.
6. The method according to claim 4, wherein the intermediate unit stores a consumption state corresponding to the data consumption unit, and the consumption state includes the first sequence number, which is used to indicate that the data consumption unit is about to consume the first data block. The providing the first data block to the data consumption unit according to the arrangement position of the first sequence number in the multiple consecutive sequence numbers includes: In response to a consumption request from the data consumption unit, determining whether the intermediate unit stores the first data block according to the consumption status; In case it is determined that the intermediate unit stores the first data block, providing the first data block to the data consumption unit; The consumption status is updated. 7 . The method according to claim 6 , wherein the determining that the data consumption unit has obtained the first data block comprises: determining that the data consumption unit has obtained the first data block according to the consumption state.
8. The method according to claim 6, in the case where the consumption status includes only the first sequence number, updating the consumption status includes updating the consumption status to a sequence number after the first sequence number; In the case where the consumption state includes a plurality of sequence numbers, updating the consumption state includes deleting the first sequence number in the consumption state.
9. The method according to claim 8, wherein the consumption status includes the first sequence number and the second sequence number, the first sequence number is smaller than the second sequence number, and the first sequence number and the second sequence number are consecutive sequence numbers, Providing the first data block to a data consumption unit according to the arrangement position of the first sequence number in the plurality of consecutive sequence numbers comprises: In response to a consumption request from the data consumption unit, determining whether the intermediate unit stores the first data block and the second data block corresponding to the second sequence number according to the consumption status; When it is determined that the intermediate unit stores the second data block but does not store the first data block, providing the second data block to the data consumption unit; The second sequence number is deleted in the consumption state.
10. According to the method described in claim 1, the first data block and its first sequence number are obtained by any data production unit through the following process: receiving initial data and its corresponding first sequence number from a data distribution unit, and obtaining the first data block and the first sequence number based on the initial data and the first sequence number.
11. A device for providing data in an orderly manner, comprising: A receiving unit, configured to receive a first data block and a first sequence number thereof from any data production unit among the plurality of data production units, wherein the first sequence number is one of a plurality of consecutive sequence numbers, and the plurality of consecutive sequence numbers respectively correspond to a plurality of data blocks generated by the plurality of data production units; A storage unit, used for storing the first data block and its first sequence number in an associated manner; A providing unit is used to provide the first data block to a data consumption unit according to the arrangement position of the first sequence number in the multiple consecutive sequence numbers.
12. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.
