Apparatus, method, and computer storage media for collecting and transmitting real-time data

Abstract

Example implementations described herein relate to a system that manages data dispatch within an Internet of Things (IoT) system, which can involve a first process to ingest new data and perform one of dispatching the new data or queuing the new data; a second process performed at a lower priority than the first process, which involves determining whether the queued data exceeds a retry count; if the retry count does not exceed a threshold, forwarding the queued data to a third process; and if the queued data exceeds the retry count, popping the queued data into an error process; and a third process performed after receiving the queued data from the second process, which involves attempting to dispatch the queued data.

Description

BACKGROUND TECHNICAL FIELD

[0002] The present disclosure relates generally to Internet of Things (IoT) systems, and more particularly, to facilitating the storage and forwarding of data flows in IoT systems. BACKGROUND

[0004] When collecting real-time sensor data and sending such data to corresponding destinations, there are two factors that impact IoT systems. One factor is ensuring that there is no data loss at the destination. Another factor is keeping the data flow path as smooth as possible.

[0005] In IoT systems, sensor data can be streamed in real-time to a processing system, such as an edge node, which needs to forward such data to appropriate destinations, such as cloud systems, analytics systems, underlying client devices, etc., depending on the desired implementation. However, due to bandwidth limitations and the amount of sensor data to be properly routed and transmitted, it can be difficult to ensure the above two factors if a large amount of data is received within the same timeframe and needs to be processed by the processing system quickly, especially if the data flow path is to be maintained in real-time or if the data is too large and intensive computing resources are needed to dispatch. SUMMARY

[0006] Example implementations described herein relate to systems and methods that implement storage and forwarding at an edge system or other processing system to enable data transmission in IoT systems. In example implementations described herein, when the data flow path is congested, when the destination is unavailable, or in other cases where data cannot be dispatched immediately, received data is stored at a local database. Example implementations then gradually thread the stored data back into the main data flow to be dispatched to its intended destination by utilizing a store-and-forward process to avoid impacting the real-time data flow path.

[0007] As will be described herein, example implementations described herein relate to a system that manages data dispatch within an Internet of Things (IoT) system, which can involve a first process for ingesting new data and performing one of dispatching the new data or queuing the new data. Through such a process, if the data is not dispatched (e.g., dispatch fails for some reason), the data is queued, which allows example implementations to pull the undispacted data out of the data stream so that it does not block the destination pipeline from dispatching data that can be dispatched, while allowing the system an opportunity to retry dispatching the undispacted data to avoid losing data at the destination.

[0008] Example implementations also relate to a second process executing at a lower priority than the first process, to determining whether the queued data exceeds a retry count; and forwarding the queued data to a third process if the retry count does not exceed a threshold. In this way, data can be automatically re-queued with updated retry counts and timestamps, as will be described herein.

[0009] Example implementations also relate to popping the queued data into an error process if the queued data exceeds the retry count. The third process can also execute after receiving the queued data from the second process, and involves attempting to dispatch the queued data. With such example implementations, queued undispacted data can be folded back into the main stream of data transfer. Such example implementations contrast with related art implementations that involve blocking a path for new data, sending the queued data, and then unblocking. In contrast, example implementations described herein involve a process of adding data to the main stream in a stepwise threading process that provides undispacted data through the pipeline normally without interrupting the flow of incoming data. In example implementations, once such undispacted data is successfully dispatched to its original intended destination, the data can be removed from the queue.

[0010] Aspects of the present disclosure can include a method that can involve: a first process involving ingesting new data and performing one of dispatching the new data or queuing the new data; a second process executing at a lower priority than the first process, involving determining whether the queued data exceeds a retry count; forwarding the queued data to a third process if the retry count does not exceed a threshold; and popping the queued data into an error process if the queued data exceeds the retry count; and a third process executing after receiving the queued data from the second process, involving attempting to dispatch the queued data.

[0011] Aspects of the present disclosure can include a computer program that can involve instructions for a first process involving ingesting new data and performing one of dispatching the new data or queuing the new data; instructions for a second process executing at a lower priority than the first process, involving determining whether the queued data exceeds a retry count; forwarding the queued data to a third process if the retry count does not exceed a threshold; and popping the queued data into an error process if the queued data exceeds the retry count; and instructions for a third process executing after receiving the queued data from the second process, involving attempting to dispatch the queued data. The instructions can be stored on a non-transitory computer readable medium and configured to be executed by one or more processors.

[0012] Aspects of the disclosure can include a system that can involve: a unit to execute a first process, the first process involving ingesting new data and performing one of dispatching the new data or queuing the new data; a unit to execute a second process performed at a lower priority than the first process, the second process involving determining whether the queued data exceeds a retry count; forwarding the queued data to a third process if the retry count does not exceed a threshold; and popping the queued data into an error process if the queued data exceeds the retry count; and a unit to execute a third process performed after receiving the queued data from the second process, the third process involving attempting to dispatch the queued data.

[0013] Aspects of the disclosure can include an apparatus that can include: a processor configured to execute a first process, the first process involving ingesting new data and performing one of dispatching the new data or queuing the new data; a second process performed at a lower priority than the first process, the second process involving determining whether the queued data exceeds a retry count; forwarding the queued data to a third process if the retry count does not exceed a threshold; and popping the queued data into an error process if the queued data exceeds the retry count; and a third process performed after receiving the queued data from the second process, the third process involving attempting to dispatch the queued data.

[0014] Through the example implementations described herein, older data can be reincorporated into an IoT system data flow even after failing to dispatch to an intended destination while avoiding disruption and minimizing impact to real-time data flow paths. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 An example visualization of a store-and-forward process for data is shown in accordance with an example implementation.

[0016] Figure 2 An example of single data forwarding is shown in accordance with an example implementation.

[0017] Figure 3 An example of batch data forwarding is shown in accordance with an example implementation.

[0018] FIGS. 4(a) through 4(c) show example flow diagrams in accordance with example implementations.

[0019] FIGS. 5(a) and 5(b) show examples of management information that can be used to implement the example implementations described herein.

[0020] Figure 6 A system involving multiple apparatuses and edge / cloud processing apparatuses is shown in accordance with an example implementation.

[0021] Figure 7 An example computing environment having example computer devices suitable for use in some example embodiments is shown. DETAILED DESCRIPTION

[0022] The following detailed description provides further details on the drawings and example embodiments of the present application. Reference signs and descriptions of redundant elements between the drawings will be omitted for the sake of clarity. The terms used throughout the present specification are provided as examples and are not intended to be limiting. For example, use of the term "automatically" can refer to a fully automatic or semi-automatic embodiment involving user or administrator control in certain aspects of the embodiment, depending on the desired embodiment of one of ordinary skill in the art practicing the embodiments of the present application. The user can make selections through a user interface or other input device, or the selections can be made through a desired algorithm. The example embodiments as described herein can be used individually or in combination, and the functions of the example embodiments can be implemented through any device depending on the desired embodiment.

[0023] Example embodiments described herein relate to implementing a store-and-forward edge / cloud process for an IoT system that involves real-time streaming of large amounts of data to multiple output locations. In such IoT systems, there is often a requirement that data cannot be lost if possible and needs to be dispatched to an output destination. In such IoT systems, it is possible to prevent dispatching data from reaching its destination due to network connectivity issues, the destination being down, the received data being malformed, and so on. However, when some received data cannot be dispatched, the system can still receive a live data stream, so the dispatch of such data should not be stopped due to a failure in dispatching some data.

[0024] In example embodiments, data messages received by the system are associated with a timestamp to implement a priority system. In example embodiments, when there is intake of new data messages, a priority is assigned to the data messages based on the time it enters the system, such that a single data message is not retried for repeated dispatching. Thus, there is a delay after retrying a data message, such that the system can gradually recover the processing to be done and allow data to be successfully sent.

[0025] Figure 1 An example visualization of a store-and-forward process for data according to example embodiments is shown. In Figure 1 In the example, a data flow and process is shown as a highway with vehicles, each vehicle representing data to be sent (e.g., packets, files, messages, etc.) for ease of explanation.

[0026] In an IoT system, a processing system or edge system receives a real-time data stream 100. Example implementations of a store-and-forward data process can be implemented in an edge system or processing system configured to dispatch data to various destinations. Such an edge system or processing system is configured to collect data that fails to reach its intended destination. The primary real-time stream path to the destination can continuously receive the data stream regardless of data path congestion or smoothing due to the real-time nature of the IoT system. Thus, the store-and-forward data process of example implementations described herein is configured to return the collected data to the primary stream to smoothly reach its intended destination without impacting the real-time data stream. In example implementations, the data stream behaves like a highway metering gate to allow local traffic (stored data) to return to the main highway traffic (the primary real-time data stream). In this way, the example implementation process eliminates data loss internally due to disconnections between the source and destination within the allowed threshold of retry counts and thereby minimizes impact on the directed real-time data stream.

[0027] In example implementations as described herein, the store-and-forward process utilizes the following administrative information to manage data stored in the queue.

[0028] Retry count indicates the number of times a particular queued data is re-dispatched from the queue to the real-time data stream before it leaves and is sent to a configured error destination for error process handling. The retry count is decremented each time a data point is retried to track when the data expires and is handled by the error process.

[0029] Retry rate is the amount of time to wait between each dispatch from the queue back to the real-time data stream, which is conceptually similar to a highway metering green light interval.

[0030] Batch count indicates the maximum number of data messages to dispatch from the queue per retry attempt, which is conceptually similar to the number of cars per green light at a highway metering gate.

[0031] Attempted timestamp is a timestamp updated at each retry attempt of queued data within the queue. This field allows the system to order for priority based on the time received within the system. When new data is received, the attempted timestamp field is set to the current time.

[0032] As Figure 1As shown, the store-and-forward process is conceptually similar to the process of a highway system, as the process handles data in a manner similar to how a highway handles cars that are attempting to merge into traffic. Incoming real-time data can be visualized as two cars entering the main highway of the system. This stream, if not congested, will continuously dispatch data to its intended destination as the data makes its way through the system. When a data point fails to dispatch or reach its intended destination at 101, the data is forwarded to the store-and-forward process, which will queue the data and later retry the dispatch through the same real-time data stream.

[0033] As shown at 102, such data is queued for an attempt to re-enter the real-time data stream. If the data point has not expired, the data is re-queued in the store-and-forward process to re-enter the real-time stream and attempt to reach its intended destination.

[0034] At 103, if the retry count reaches zero, the data exits the highway from the queue due to expiration. Each data point contains a retry count, which dictates when the data point expires. When the retry count reaches zero, it is taken out of the queue and dispatched to a configured error destination, such as an error queue or error handling process set according to the desired implementation. The retry count acts as a threshold gate to allow the user to manipulate how long a data point can be stored in the queue before being kicked out. Depending on the retry count set according to the desired implementation, the size of the queue will be adjusted and allocated accordingly.

[0035] When processing queued messages and dispatching to the real-time data stream, there can be two settings that can be utilized according to the desired implementation, which can be single data forwarding and batch data forwarding.

[0036] Figure 2 An example of single data forwarding according to an example implementation is shown. When set for single data forwarding, the store-and-forward process dispatches messages from its queue one at a time. In Figure 2 In the example shown, DP is an abbreviation for Data Point and is used to highlight each data point as it is received or queued; RC is an abbreviation for Remaining Retry Count and is used to represent the retry count of the current DP as it counts down. The ingestion rate is the current rate at which new data is being received to be queued and dispatched via the store-and-forward process. As Figure 2 As shown, the following settings are configured for the store-and-forward process:

[0037] Ingestion rate - 1 DP / second

[0038] Retry rate - 1 DP / second

[0039] Retry count - 2

[0040] Batch Count - 1

[0041] As the time interval steps from 1 second - 2 seconds - 3 seconds - n seconds, the visualization of the queue shows how incoming data is added to the queue and how retried data is moved to lower priority within the queue. At each time interval, the system retrieves the current queue sorted in ascending order by the timestamp field of the attempt, and selects the first DP to retry. At each retry attempt of the DP, the store-and-forward process decrements the RC, and passes the DP to the bottom of the queue by updating the timestamp field of the attempt with the current time to allow the previously stored DP to be retried with higher priority. Each time a DP is taken from the queue, the system checks the RC value, and if it equals zero (i.e., no more retry attempts remain and expire), the data is dispatched to an error handling location (e.g., an error topic queue). After that, the next value at the top of the queue is immediately retried and follows the same process. If the DP has a valid RC, an alert is sent to the system to notify the service to retry dispatching the DP to its intended destination. When new data is queued in the buffer that implements the queue, the queued data is appended to the bottom of the stack with the lowest priority of the current system by setting the timestamp of the attempt to the current time.

[0042] Figure 3 An example of batch data forwarding is shown in accordance with example embodiments. When set for batch data forwarding, the store-and-forward process dispatches a maximum value of N messages at a time. In Figure 3 DP is an abbreviation for data point and is used to highlight each data point as it is received or queued; RC is an abbreviation for remaining retry count and is used to represent the retry count of the current DP as it counts down. The ingestion rate is the current rate at which new data is received to be queued and dispatched via the store-and-forward process. As Figure 3 shown, the following settings are configured for the store-and-forward process:

[0043] Ingestion Rate - 1 DP / second

[0044] Retry Rate - 1 DP / second

[0045] Retry Count - 2

[0046] Batch Count - 2

[0047] The queue visualization shows how incoming data is added to the queue, and how retried data is moved to lower priority within the queue, stepping from 1 second to 2 seconds to 3 seconds to n seconds. For each time interval, the system retrieves the current queue sorted in ascending order by the timestamp field of the attempts and selects the top set of DPs to be retried based on the batch count. During retrying for each DP set, the RC is decremented for each data point, and the DP is moved to the bottom of the queue by updating the timestamp field of the attempts with the current time, allowing previously stored DPs to be retried with higher priority. Whenever a DP set is removed from the queue, the system checks each RC value, and if any of its values ​​are equal to zero (i.e., no more retry attempts remain and the queue has expired), those particular DPs are assigned to an error disposal location (in our case, the error topic queue). Afterward, if there are remaining DPs with valid RCs, an alert is sent to the system to notify the service to retry those DPs to their intended destinations. If there are no remaining DPs, the same process is used to immediately retry the next batch at the top of the queue. When new data is queued during storage and forwarding, it is appended to the bottom of the stack with the lowest priority in the current system by setting the timestamp of the attempt to the current time.

[0048] Figures 4(a) and 4(c) illustrate example flowcharts according to an example implementation. Specifically, Figure 4(a) shows a first process for managing the ingestion of new data executed when data is received at 401. At 402, the first process ingests the new data to determine whether the data can be assigned to a destination. If so (yes), the process proceeds to 403 to assign the data to the corresponding destination; otherwise (no), the process proceeds to 404 to queue the data in a database or temporary storage according to the desired implementation. Furthermore, the process at 404 may also involve causing the first process to trigger a second process for implementing store-and-forward functionality according to the desired implementation.

[0049] Figure 4(b) illustrates a second process for implementing the store-and-forward function according to an example implementation. In the example implementation, the store-and-forward process is executed when a triggering event has occurred at 410. According to a desired implementation, such triggering events may include: when a period of time has elapsed, when the first process calls the second process to ensure retry rate, when an error handling process is executed, etc. In the example implementation where the first and second processes are threaded processes, the second process is called with a lower priority than the first process to ensure that the ingestion of new data takes precedence over the execution of the store-and-forward process.

[0050] At 411, the second process checks the database to determine if there is any remaining queued data. At 412, it is determined whether such queued data exists. If not (no), the second process ends; otherwise (yes), the process proceeds to 413. At 413, it is determined whether the retry count associated with the queued data has not yet been exceeded (e.g., the data has not yet expired). In the example implementation described above, the retry count is implemented by decrementing the retry count each time queued data fails to be dispatched, in which case it is determined that the retry count has been exceeded when the count drops to zero. However, those skilled in the art can also change this implementation to incrementing the retry count each time queued data fails to be dispatched, in which case it is determined that the retry count has been exceeded when the count reaches a certain threshold. Any such implementation can be utilized depending on the desired implementation. If the retry count has been exceeded (yes), the process proceeds to 415 to perform an error handling process by removing the queued data from the database and entering an error process, thereby popping the queued data from the database. According to the desired implementation, such an error handling procedure can take the form of a threaded procedure configured to remove the popped data and send a message back to the source of the popped data to indicate that data transmission has failed, or otherwise. According to the desired implementation, since no data was queued in the original iteration due to exceeding the retry count, the execution of the error handling procedure at 415 can also serve as a trigger for re-executing the second procedure at 410. In this way, the second procedure can be re-executed after the error handling procedure 415 to act on the next queued data, ensuring that the second procedure is repeated continuously until at least some queued data that has not exceeded the retry count can be processed at least once.

[0051] If the retry count has not been exceeded (No), the process proceeds to 414 to perform the third process shown in Figure 4(c), which is the egress process for the queued data. At 416, the process proceeds to 416 to decrement the retry count associated with the queued data, and then at 417, the data with the updated timestamp is requeued.

[0052] Figure 4(c) illustrates an example of the third process implementing outflow process 414 according to an exemplary embodiment. The third process 414 is triggered at 420 when queued data is received. At 421, the third process attempts to dispatch the data through the real-time data stream. If dispatch is successful (yes), the queued data is dispatched from the system at 422, thereby removing the data from the queue by the third process. In the exemplary embodiment, the processes of Figures 4(a) through 4(c) can be executed as threads, such that the flow of the second process at 416 and 417 can be executed concurrently by the second process while the third process attempts to dispatch data. Therefore, the third process is configured to handle the removal of data from the queue once data has been successfully dispatched, taking into account data that has been requeued by the second process even if it has already been dispatched.

[0053] Figures 5(a) and 5(b) illustrate examples of management information that can be used to implement the exemplary embodiments described herein. Specifically, Figure 5(a) illustrates example management information for system management of different types of sensors according to an exemplary embodiment. In the exemplary embodiments herein, the IoT system can manage different types of sensors, which may be associated with different sizes of data, etc. As shown in Figure 5(a), the management information may include sensor type, data size, retry count, batch flag, and guaranteed rate. Sensor type indicates the type of sensor associated with the received data. Depending on the desired implementation, the sensor type may be determined based on the source sensor array, by the data size, or otherwise. Data size indicates the size of the data message associated with the sensor type. Retry count indicates the retry count set for the data message of this sensor type. The retry count can be set according to the data size to implement the desired implementation (e.g., a larger retry count for smaller data, a smaller retry count for larger data, etc.). Batch count flag indicates whether sensor data of this type can be transmitted in batches.

[0054] In the example implementation, based on the management information in Figure 5(a) and the type of sensor data stored in the queue, the system's first, second, and third processes can be configured to handle the queued data accordingly. For example, if all the queued data is associated with a sensor type in which batch dispatch is enabled (yes), these processes can be configured to dispatch messages in batches (e.g., attempt to batch dispatch data messages until a size such as 50MB is dispatched). A guaranteed rate can be set based on the data size (e.g., based on the size of the data being sent, the rate is set to X attempts per second, where smaller data can be retried more frequently than larger data).

[0055] Figure 5(b) illustrates example management information for data stored in a database queue according to an example implementation. The data may be associated with metadata indicating a data message ID, source, destination, data size, sensor type, and retry count. The data message ID is an identifier associated with the data when it is ingested into the system. According to a desired implementation, the data message ID may be a timestamp provided by the system when data ingestion occurs. In this way, the data message ID can be updated with a new timestamp whenever a data message is requeued into the database. The source indicates the origin of the data message and may take any form (e.g., IP address, sensor ID, etc.) according to a desired implementation. When the system needs to handle errors in the data, the data message may be transmitted to the source to indicate that the message failed to be dispatched to the destination. The destination may indicate the output destination of the data message and may take any form (e.g., IP address, etc.) according to a desired implementation. The data size may indicate the size of the data message. The sensor type may indicate the type of sensor data received. The retry count is the retry count associated with the data message.

[0056] Figure 6 A system involving multiple devices and an edge / cloud processing device according to an exemplary embodiment is illustrated. One or more devices or device systems 601-1, 601-2, 601-3, and 601-4 are communicatively coupled to a network 600 and transmit sensor data to an edge / cloud processing device 602, which is configured to receive such data and transmit it to its corresponding destination. The edge / cloud processing device 602 manages a database 603 that manages queued data to be dispatched. Such devices or device systems may include stationary devices or equipment, such as coolers, air conditioners, servers, and mobile devices or equipment, such as cars, trucks, cranes, and any other devices capable of transmitting live streaming data. Such devices may involve sensors that provide sensor data to a maintenance planning device 602.

[0057] Figure 7 An example computing environment with example computer devices suitable for use in some example implementations is shown, such as Figure 6 The edge / cloud processing device 602 is shown. The computer device 705 in the computing environment 700 may include one or more processing units, cores or processors 710, memory 715 (e.g., RAM, ROM, etc.), internal storage devices 720 (e.g., magnetic, optical, solid-state storage, and / or organic storage devices), and / or I / O interfaces 725, any of which may be coupled to a communication mechanism or bus 730 for transmitting information or embedded in the computer device 705. According to a desired implementation, the I / O interface 725 is also configured to receive images from a camera or provide images to a projector or display.

[0058] Computer device 705 may be communicatively coupled to input / user interface 735 and output device / interface 740. One or both of input / user interface 735 and output device / interface 740 may be wired or wireless interfaces and may be detachable. Input / user interface 735 may include any device, component, sensor, or interface (physical or virtual) that can be used to provide input (e.g., buttons, touchscreen interface, keyboard, pointing / cursor controls, microphone, camera, Braille, motion sensor, optical reader, etc.). Output device / interface 740 may include a display, television, monitor, printer, speaker, or Braille, etc. In some example embodiments, input / user interface 735 and output device / interface 740 may be embedded in or physically coupled to computer device 705. In other example embodiments, other computer devices may serve as or provide the functionality of input / user interface 735 and output device / interface 740 of computer device 705.

[0059] Examples of computer device 705 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablet computers, laptop computers, personal computers, portable televisions, wireless devices, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, kiosks, televisions with one or more processors embedded therein and / or coupled thereto, wireless devices, etc.).

[0060] Computer device 705 may be communicatively coupled (e.g., via I / O interface 725) to external storage device 745 and network 750 for communication with any number of networking components, devices, and systems, including one or more computer devices with the same or different configurations. Computer device 705 or any connected computer device may be used to provide services as a server, client, thin server, general-purpose machine, special-purpose machine, or otherwise labeled, or referred to as a server, client, thin server, general-purpose machine, special-purpose machine, or other label.

[0061] I / O interface 725 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocol, etc.) for transmitting information to and / or transmitting information from at least all connected components, devices, and networks in computing environment 700. Network 750 may be any network or combination of networks (e.g., Internet, local area network, wide area network, telephone network, cellular network, satellite network, etc.).

[0062] Computer device 705 may communicate using and / or utilizing computer-usable or computer-readable media, including transient and non-transient media. Transient media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transient media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video disks, Blu-ray discs), solid-state media (e.g., RAM, ROM, flash memory, solid-state memory), and other non-volatile storage devices or memories.

[0063] Computer device 705 can be used to implement technologies, methods, applications, processes, or computer-executable instructions in some example computing environments. It is capable of retrieving computer-executable instructions from transient media, storing computer-executable instructions on non-transient media, and retrieving computer-executable instructions from such media. The executable instructions can originate from one or more of any programming language, scripting language, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0064] One or more processors 710 can execute under any operating system (OS) (not shown) in a local or virtual environment. One or more applications can be deployed, including a logic unit 760, an application programming interface (API) unit 765, an input unit 770, an output unit 775, and an inter-unit communication mechanism 795 for communication between different units, with the OS, and with other applications (not shown). The described units and elements may vary in design, function, configuration, or implementation, and are not limited to the description provided. One or more processors 710 may be in the form of a hardware processor, such as a central processing unit (CPU) or a combination of hardware and software units.

[0065] In some example implementations, when API unit 765 receives information or execution instructions, the information or execution instructions can be transmitted to one or more other units (e.g., logic unit 760, input unit 770, output unit 775). In some cases, logic unit 760 can be configured to control the information flow between control units and direct the services provided by API unit 765, input unit 770, and output unit 775 (in some example implementations described above). For example, the flow of one or more processes or implementations can be controlled by logic unit 760 alone or together with API unit 765. Input unit 770 can be configured to receive input for the calculations described in the example implementations, and output unit 775 can be configured to provide output based on the calculations described in the example implementations.

[0066] The memory 715 can be configured to store the management information shown in Figures 5(a) and 5(b) to implement the example implementation. The database 602 can be implemented in the form of a storage system by some combination of the memory 715 and / or internal storage device 720 and external storage device 745 to implement the example implementation according to the desired implementation.

[0067] One or more processors 710 may be configured to execute a first process involving ingesting new data, as shown in FIG4(a); and to execute one of dispatching new data or queuing new data; a second process executed at a lower priority than the first process, as shown in FIG4(b), involving determining whether the queued data exceeds a retry count; forwarding the queued data to a third process if the retry count does not exceed a threshold; and popping the queued data into an error process if the queued data exceeds the retry count; and a third process executed after receiving the queued data from the second process, as shown in FIG4(c), involving attempting to dispatch the queued data. In an example implementation where the first, second, and third processes are executed as threads, the second process may be periodically invoked at a lower priority than the first process (e.g., the first process is executed continuously in real time, and the second process is executed every 2-3 seconds) to prioritize newly introduced data over queued data during data dispatch.

[0068] In the example implementation, the second process may also involve requeuing the queued data after forwarding it to the third process, as shown at 417, whereby if the third process successfully dispatches the data, it is responsible for deleting the queued data and removing the entry from the management information in FIG5(b). In this way, the second and third processes can be performed simultaneously. In the example implementation, requeuing the queued data after forwarding it to the third process may involve decrementing the retry count of the queued data at 416; and requeuing the queued data with an updated timestamp and a decremented retry count at 417. In this way, the management information in FIG5(b) can be updated simultaneously with the attempted dispatch process from the third process in FIG4(c).

[0069] In an example implementation, one or more processors 710 may be configured to re-execute the second process for the next queue of data after an error process pops queued data. In this example implementation, if the second process is invoked and only the error handling process 415 is invoked from the execution of the second process, the second process can be repeated to ensure that the outflow process 414 is performed for some queued data when the second process is invoked.

[0070] As shown in Figure 5(a), new data can be received from one of several different types of sensors, wherein the first, second, and third processes are configured to be performed based on one of the several different types of sensors associated with the new data, by adjusting the retry count, the size of the data to be dispatched, etc., according to a desired implementation. In such an example implementation, data of various sizes and data from various types of sensors can be processed by one or more processors 710 and need not be consistently implemented by a store-and-forward process.

[0071] In an example implementation, one or more processors 710 can queue new data 404 by setting a retry count for the new data based on the size of the new data. The retry count can be set according to management information as shown in FIG5(a), which allocates the retry count based on the size of the data (e.g., smaller data has a higher retry count).

[0072] According to the desired implementation, the second process can be executed at a guaranteed retry rate set based on the size of the new data. For example, if the size of each data message in the queue does not exceed a certain size, the second process can be executed periodically at a rate of one message per second, and this rate can be adjusted upwards or downwards according to the desired implementation.

[0073] According to the desired implementation, the second process can be performed based on a batch count set according to the data size. For example, the second process can be performed periodically to attempt to dispatch a set number of messages, or as many messages as possible, until the total size of the data has been dispatched. These messages may become more numerous for smaller data or less numerous for larger data, or take other forms depending on the desired implementation.

[0074] This section presents some detailed descriptions of the algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means by which those skilled in the art of data processing communicate the essence of their innovations to others. An algorithm is a series of defined steps that lead to a desired final state or result. In example implementations, the steps performed require the physical manipulation of tangible quantities to obtain tangible results.

[0075] Unless otherwise specifically stated, it should be understood, as is apparent from the discussion, that throughout the specification, the use of terms such as “processing,” “operation,” “calculation,” “determine,” “display” can include the actions and processes of a computer system or other information processing device, wherein the computer system or other information processing device manipulates data represented as physical (electronic) quantities in the registers and memories of the computer system and converts them into other data similarly represented as physical quantities in the memory or registers or other information storage devices, transmission or display devices of the computer system.

[0076] The exemplary embodiments may also relate to means for performing the operations described herein. This means may be specifically constructed for the desired purpose, or it may include one or more general-purpose computers that can be selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in computer-readable media such as computer-readable storage media or computer-readable signal media. Computer-readable storage media may relate to tangible media, such as, but not limited to, optical discs, magnetic disks, read-only memory, random access memory, solid-state devices and drives, or any other type of tangible or non-transient media suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays mentioned herein are not inherently related to any particular computer or other device. The computer program may relate to a pure software implementation that relates to instructions for performing the operations of the desired implementation.

[0077] Based on the examples in this document, various general-purpose systems, programs, and modules can be used, or it can be demonstrated that more specialized devices can be constructed to perform the desired method steps. Furthermore, no specific programming language is referenced in the description of the example implementations. It should be recognized that various programming languages ​​can be used to implement the teachings of the example implementations described herein. The instructions of the programming language can be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0078] As those skilled in the art will recognize, the above operations can be performed by hardware, software, or a combination of software and hardware. Various aspects of the example embodiments can be implemented using circuitry and logic devices (hardware), while others can be implemented using instructions stored on a machine-readable medium (software), which, if executed by a processor, will cause the processor to perform methods implementing the embodiments of this application. Furthermore, some example embodiments of this application can be implemented solely in hardware, while others can be implemented solely in software. Additionally, the various functions described can be performed in a single unit or distributed across multiple components in any number of ways. When executed by software, the methods can be executed by a processor such as a general-purpose computer based on instructions stored on a computer-readable medium. Instructions can be stored on the medium in compressed and / or encrypted formats if desired.

[0079] Furthermore, other embodiments of the invention will become apparent to those skilled in the art upon consideration of the detailed description and practice of the teachings of this application. Various aspects and / or components of the described exemplary embodiments may be used individually or in any combination. It is contemplated that the detailed descriptions and exemplary embodiments should be considered merely as examples, and the true scope and spirit of the invention are indicated by the appended claims.

Claims

1. A method for collecting and transmitting real-time data, comprising: The first process includes: New data intake; Perform one of the following: dispatch the new data or queue the new data; A second process, executed with a lower priority than the first process, includes: Determine if the queued data exceeds the retry count; If the retry count does not exceed the threshold, the queued data is forwarded to the third process; and If the queued data exceeds the retry count, the queued data is popped into the error process; and A third process is executed after receiving queued data from the second process, the third process comprising: Attempt to distribute the queued data; The second process further includes: re-queuing the queued data after forwarding it to the third process; The process of re-queuing the queued data after forwarding it to the third process includes: Update the timestamp of the queued data; and Decrease the retry count of the queued data; and The queued data is re-queued using the updated timestamp and the decremented retry count; The second process is performed using a guaranteed retry rate set based on the size of the new data, or a batch count set based on the data size; the retry rate is the amount of time spent waiting between each dispatch from the queue back to the real-time data stream, and the batch count indicates the maximum number of data messages that each retry attempt to dispatch from the queue.

2. The method according to claim 1, further comprising: After popping the queued data into the error process, the second process is re-executed for the next queued data.

3. The method according to claim 1, wherein, The new data is received from one of a plurality of different types of sensors, wherein the first process, the second process, and the third process are configured to be performed based on the one of the plurality of different types of sensors associated with the new data.

4. The method according to claim 1, wherein, Queuing the new data includes setting a retry count for the new data based on the size of the new data.

5. A computer storage medium storing thereon instructions for executing a process, the instructions comprising: The first process includes: New data intake; Perform one of the following: dispatch the new data or queue the new data; A second process, executed with a lower priority than the first process, includes: Determine if the queued data exceeds the retry count; If the retry count does not exceed the threshold, the queued data is forwarded to the third process; and If the queued data exceeds the retry count, the queued data is popped into the error process; and A third process is executed after receiving queued data from the second process, the third process comprising: Attempt to distribute the queued data; The second process further includes: re-queuing the queued data after forwarding it to the third process; The process of re-queuing the queued data after forwarding it to the third process includes: Update the timestamp of the queued data; and Decrease the retry count of the queued data; and The queued data is re-queued using the updated timestamp and the decremented retry count; The second process is performed using a guaranteed retry rate set based on the size of the new data, or a batch count set based on the data size; the retry rate is the amount of time spent waiting between each dispatch from the queue back to the real-time data stream, and the batch count indicates the maximum number of data messages that each retry attempt to dispatch from the queue.

6. The computer storage medium according to claim 5, further comprising: After popping the queued data into the error process, the second process is re-executed for the next queued data.

7. The computer storage medium according to claim 5, wherein, The new data is received from one of a plurality of different types of sensors, wherein the first process, the second process, and the third process are configured to be performed based on the one of the plurality of different types of sensors associated with the new data.

8. The computer storage medium according to claim 5, wherein, Queuing the new data includes setting a retry count for the new data based on the size of the new data.

9. An apparatus for collecting and transmitting real-time data, comprising: The processor is configured to execute: The first process includes: New data intake; Perform one of the following: dispatch the new data or queue the new data; A second process, executed with a lower priority than the first process, includes: Determine if the queued data exceeds the retry count; If the retry count does not exceed the threshold, the queued data is forwarded to the third process; and If the queued data exceeds the retry count, the queued data is popped into the error process; and A third process is executed after receiving the queued data from the second process, the third process comprising: Attempt to distribute the queued data; The second process further includes: re-queuing the queued data after forwarding it to the third process; The process of re-queuing the queued data after forwarding it to the third process includes: Update the timestamp of the queued data; and Decrease the retry count of the queued data; and The queued data is re-queued using the updated timestamp and the decremented retry count; The second process is performed using a guaranteed retry rate set based on the size of the new data, or a batch count set based on the data size; the retry rate is the amount of time spent waiting between each dispatch from the queue back to the real-time data stream, and the batch count indicates the maximum number of data messages that each retry attempt to dispatch from the queue.

Images