A real-time data statistics method and system
Through real-time data statistics method, the time period comparison mechanism of message queues and buffer zones is used to solve the problems of power statistical lag and high system load in the existing technology, real-time power consumption statistics and load reduction are achieved.
Patent Information
- Application Number
- CN202210340719.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-03-31
AI Technical Summary
In the prior art, the power statistics method has lag, resulting in a high system load and a large amount of data need to be processed.
Real-time data statistics method is adopted to obtain the device's reported data and collection time through message queues, perform time period comparison, store data to the corresponding database and cache area, realize real-time statistics and reduce system load.
Real-time statistics on electricity consumption is realized, reducing the need for the statistical system to process all data in one statistical cycle, and effectively reducing the system load.
Smart Images

Figure CN114722059B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of Internet of Things instruments and meters, and in particular to a real-time data statistics method and system. Background Art
[0002] There are a large number of devices with data collection and metering capabilities in the Internet of Things (such as electricity collection devices, water collection devices, or gas collection devices, etc.). These devices also have remote communication functions and can upload the instantaneous value of device usage data to the statistical platform in a time sequence. Taking electricity metering as an example, the electricity metering method of the existing statistical platform generally uses the cumulative amount of electricity as the data source. When the statistical platform needs to count the hourly, daily, monthly, and annual electricity consumption of the electricity collection equipment as the basis for energy consumption statistical analysis, the traditional statistical method mostly uses timed statistics. When one hour of electricity consumption is used as the basis for energy consumption analysis, the current hour counts the electricity consumption of the previous hour. This method has a lag in electricity statistics, and the electricity statistical task needs to process a large amount of data within a statistical cycle. At this time, the system load will be relatively high. Summary of the invention
[0003] Therefore, the technical problem to be solved by the present invention is to overcome the defects of the prior art that the electricity statistics method has a lag, and the electricity statistics task needs to process a large amount of data within a statistical cycle, resulting in a high system load, thereby providing a real-time data statistics method and system.
[0004] According to a first aspect, an embodiment of the present invention discloses a real-time data statistics method, including: obtaining device reported data and corresponding collection time pre-stored in a message queue, and forming at least first reported data and second reported data according to the reported data and the corresponding collection time; obtaining the second reported data and the corresponding collection time from the message queue and obtaining the first reported data and the corresponding collection time from a cache area; comparing the collection time corresponding to the first reported data with the collection time corresponding to the second reported data; when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, storing the second reported data in a first storage statistical period of a database, and writing the second reported data and the corresponding collection time into the cache area; when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the device data in a second storage statistical period of a database, and writing the second reported data and the corresponding collection time into the cache area.
[0005] Optionally, the stored statistical period data includes: device unique number, date, statistical period, starting value, ending value, starting value collection time, and ending value collection time.
[0006] Optionally, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, the second reported data is stored in the first storage statistical period of the database, including: using the end value of the second reported data stored in the first storage statistical period to replace the end value of the first reported data statistics in the current statistical period, the starting value remains unchanged, and the second reported data and the corresponding collection time are written into the cache area to update the cache area data.
[0007] Optionally, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, the second reported data is stored in the second storage statistical period of the database, including: using the acquired end value of the first reported data stored in the first storage statistical period as the starting value of the second storage statistical period, and the end value of the second reported data stored in the second storage statistical period as the end value of the second storage statistical period, the starting value and the end value of the first storage statistical period remain unchanged, and the second reported data and the corresponding collection time pair are written into the cache area, and the cache area data is updated.
[0008] Optionally, before obtaining the device reported data and the corresponding collection time pre-stored in the message queue, and forming at least first reported data and second reported data based on the reported data and the corresponding collection time, the method also includes: obtaining the first reported data and the corresponding collection time from the message queue for the first time, forming at least a first storage statistical period in the database, and writing the first reported data and collection time obtained for the first time into a cache area.
[0009] Optionally, the types of storage statistics time periods include: storage statistics hourly time period, storage statistics daily time period, storage statistics weekly time period, storage statistics monthly time period, and storage statistics annual time period.
[0010] Optionally, before obtaining the device reported data and the corresponding collection time pre-stored in the message queue, and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time, the method also includes: obtaining the first reported data and the corresponding collection time from the message queue for the first time, forming at least two types of storage statistics hourly period, storage statistics daily period, storage statistics weekly period, storage statistics monthly period, and storage statistics annual period in the database, and forming corresponding first storage statistics period in different types of storage statistics period, respectively, and writing the first reported data and collection time obtained for the first time into the cache area.
[0011] Optionally, the device reported data includes: a device unique number and a table code value.
[0012] Optionally, the reported data includes: any one of: water metering equipment and water consumption, gas metering equipment and gas consumption, and electricity metering equipment and electricity consumption.
[0013] According to the second aspect, an embodiment of the present invention further discloses a real-time data statistics system, which executes the real-time data statistics method as described in the first aspect or any optional implementation method of the first aspect, and the system includes: a message queue module, a real-time statistics service program module, a database module and a cache module, wherein the real-time statistics service program module obtains the device reported data and the corresponding collection time from the message queue module, and compares and judges the time period with the reported data and the corresponding collection time in the cache, updates the data statistics according to the database statistics rules of the judgment processing results, and updates the latest reported data and the corresponding collection time in the cache module.
[0014] The technical solution of the present invention has the following advantages:
[0015] The real-time data statistics method provided by the present invention comprises: obtaining the device reported data and the corresponding collection time pre-stored in the message queue, and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time; obtaining the second reported data and the corresponding collection time from the message queue and obtaining the first reported data and the corresponding collection time from the buffer area; comparing the collection time corresponding to the first reported data with the collection time corresponding to the second reported data; when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, storing the second reported data in the first storage statistical period of the database, and writing the second reported data and the corresponding collection time to the buffer area; when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the device data in the second storage statistical period of the database, and writing the second reported data and the corresponding collection time to the buffer area. The method of the present invention is a streaming real-time data statistics method based on time series, and a piece of data is processed in real time when a piece of data reported by a device is obtained, and the power consumption of the current statistical period can be counted in real time. At the same time, the real-time data statistics task does not need to count all the data in a statistical period, which effectively reduces the load of the statistical system. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1A flowchart of a specific example of a real-time data statistics method in an embodiment of the present invention;
[0018] Figure 2 A schematic diagram of a specific example of a real-time data statistics system in an embodiment of the present invention;
[0019] Figure 3 A schematic diagram of a specific example of a real-time data statistics method in an embodiment of the present invention;
[0020] Figure 4 A schematic diagram of a specific example of a real-time data statistics method in an embodiment of the present invention;
[0021] Figures 5A to 5D A schematic diagram of a specific example of a real-time data statistics method in an embodiment of the present invention;
[0022] Figure 6 is a principle block diagram of a specific example of a real-time data statistics device in an embodiment of the present invention;
[0023] Figure 7 FIG. 4 is a specific example diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0026] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0027] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0028] The embodiment of the present invention discloses a real-time data statistics method, which can be applied to Internet of Things devices. The real-time data includes: any one of water consumption, gas consumption and electricity consumption, and can also be extended to statistics of other cumulative quantities. In the embodiment of the present application, the real-time data statistics method recorded in the embodiment is described by taking the electricity consumption of each hour as an example. Figure 1 As shown, the method comprises the following steps:
[0029] Step 101: Acquire device reported data and corresponding collection time pre-stored in a message queue, and form at least first reported data and second reported data according to the reported data and the corresponding collection time.
[0030] Exemplarily, the device reported data and the corresponding first collection time are stored in the message queue, the second reported data is the data reported by the current device, and the first reported data is the last reported data by the device.
[0031] Step 102: Acquire the second reported data and the corresponding collection time from the message queue and acquire the first reported data and the corresponding collection time from the buffer area.
[0032] Exemplarily, the cache area stores a piece of data reported by the device (first reported data) and a corresponding collection time. Specifically, the last processed data stored in the cache area may also be referred to as cached data.
[0033] Step 103: compare the collection time corresponding to the first reported data with the collection time corresponding to the second reported data.
[0034] Step 104, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, the second reported data and the corresponding collection time are stored in the first storage statistical period of the database, and the second reported data and the corresponding collection time are written into the cache area.
[0035] Exemplarily, the statistical period is a time period that can be pre-set according to statistical requirements. This specific embodiment does not limit the value of the statistical period. The corresponding collection time of the data reported by the current device (the second reported data) and the corresponding collection time of the data reported by the previous device (the first reported data) belong to the same statistical period. The second reported data is stored in the first storage statistical period of the database, and the data stored by the current device (the second reported data) and the corresponding collection time are stored in the cache area.
[0036] Step 105, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, the device data is stored in the second storage statistical period of the database, and the second reported data and the corresponding collection time are written into the cache area.
[0037] Exemplarily, if the collection time corresponding to the data reported by the current device (the second reported data) and the collection time corresponding to the data reported by the previous device (the first reported data) do not belong to the same statistical period, the second reported data will be stored in the second storage statistical period of the database, and the data stored by the current device (the second reported data) and the corresponding collection time will be stored in the cache area.
[0038] The real-time data statistics method provided by the present invention is a streaming real-time data statistics method based on time series. When a piece of data reported by a device is obtained, the data is processed in real time. The power consumption of the current statistical period can be counted in real time. At the same time, the real-time data statistics task does not need to count all the data within a statistical period, which effectively reduces the load of the statistical system.
[0039] As an optional implementation manner of the present invention, the stored statistical period data includes: a unique device number, a date, a statistical period, a starting value, an ending value, a starting value collection time, and an ending value collection time.
[0040] As an optional implementation manner of the present invention, the data reported by the device includes: a unique device number and a table code value.
[0041] Exemplarily, taking the statistics of electricity consumption of an electric meter as an example, the statistical period can be hours, days, months, and years; in this specific real-time example, the statistical period is hours, and the corresponding collection time of the data reported by the current device (the second reported data) and the corresponding collection time of the data reported by the previous device (the first reported data) belong to the same statistical period, then the currently processed statistical data is stored in the first storage statistical period of the database. The data stored in the first storage statistical period include the device unique number, date, statistical period, starting value, end value, starting value collection time, end value collection time, and device reported data. The device reported data includes a meter code value and a device unique number. The meter code data is an instantaneous value, which is the electric energy reading displayed by the meter at the corresponding collection time. The value is equal to the sum of the electricity consumption of the meter from the beginning of use to the current time. The data collected and reported by the device are both meter code data. The electricity consumption is calculated based on the meter code data. The device unique number is used to distinguish the device data reported by different devices. In this method, the device data reported by different devices can be stored in different pre-set storage areas, or stored in the same storage area and the device identifier can be used to distinguish the device data reported by different devices. When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data are within the same hour, the second reported data currently being processed is stored in the first storage statistical period of the database, and the second reported data and the corresponding collection time are stored in the cache area. When it is necessary to count the power consumption of the current hour, the power consumption is equal to the table code data corresponding to the second reported data minus the starting table code data; when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same hour, the data of the statistical period in which the collection time corresponding to the second reported data of the device is located is added to the database (the second storage statistical period), the starting table code is the table code data corresponding to the second collection time, and the ending table code is the table code data corresponding to the first collection time.
[0042] As an optional implementation of the present invention, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, the second reported data is stored in the first storage statistical period of the database, including: using the acquired end value of the second reported data stored in the first storage statistical period to replace the end value of the first reported data statistics in the current statistical period, the starting value remains unchanged, and the second reported data and the corresponding collection time are written into the cache area to update the cache area data.
[0043] Exemplarily, when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, the table code value corresponding to the first reported data last stored in the first storage period of the database (the last stored value) is replaced with the table code value corresponding to the current processing data (the second reported data), and the data in the cache area is updated using the first reported data and the corresponding collection time. Specifically, when it is necessary to count the power consumption of the current hour, the power consumption is equal to the table code data corresponding to the second reported data minus the starting table code data.
[0044] As an optional implementation manner of the present invention, the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, and the second reported data is stored in the second storage statistical period of the database, including: using the acquired end value of the first reported data stored in the first storage statistical period as the starting value of the second storage statistical period, and the end value of the second reported data stored in the second storage statistical period as the end value of the second storage statistical period, the starting value and the end value of the first storage statistical period remain unchanged, and the second reported data and the corresponding collection time pair are written into the cache area, and the cache area data is updated.
[0045] Exemplarily, the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical time period, and the data of the statistical time period in which the collection time corresponding to the second reported data of the device is located (the second stored statistical time period) is added to the database, the starting table code is the table code data corresponding to the first reported data, and the ending table code is the table code data corresponding to the second reported data, and the second reported data and the corresponding collection time pair are written into the cache area, and the cache area data is updated.
[0046] As an optional implementation of the present invention, before obtaining the device reported data and the corresponding collection time pre-stored in the message queue, and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time, the method also includes: obtaining the first reported data and the corresponding collection time from the message queue for the first time, forming at least a first storage statistical period in the database, and writing the first reported data and the collection time obtained for the first time into a cache area.
[0047] As an optional implementation manner of the present invention, types of storage statistics time periods include: storage statistics hourly time period, storage statistics daily time period, storage statistics weekly time period, storage statistics monthly time period, and storage statistics annual time period.
[0048] As an optional embodiment of the present invention, before obtaining the device reported data and the corresponding collection time pre-stored in the message queue, and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time, the method also includes: obtaining the first reported data and the corresponding collection time from the message queue for the first time, forming at least two types of storage statistics hourly period, storage statistics daily period, storage statistics weekly period, storage statistics monthly period, and storage statistics annual period in the database, and forming corresponding first storage statistics period in different types of storage statistics period, and writing the first reported data and collection time obtained for the first time into the cache area.
[0049] Exemplarily, if the data currently being processed is the device data reported by the device for the first time (first reported data), then the cache data does not exist at this time, the starting table code is the table code data of the current device data, the data reported by the current device is stored in the cache area, and the current power consumption = starting table code data - table code data of the current device data = 0. The statistical time periods stored in the database include the hourly time period for storage statistics, the daily time period for storage statistics, the weekly time period for storage statistics, the monthly time period for storage statistics, and the annual time period for storage statistics. When the device reports data for the first time, at least two of the hourly time period for storage statistics, the daily time period for storage statistics, the weekly time period for storage statistics, the monthly time period for storage statistics, and the annual time period for storage statistics are formed in the database according to the first reported data reported by the device and the corresponding collection time.
[0050] As an optional implementation manner of the present invention, the reported data includes: any one of a water metering device and water consumption, a gas metering device and gas consumption, and an electricity metering device and electricity consumption.
[0051] The embodiment of the present invention also discloses a real-time data statistics system. Figure 2 As shown, the system includes: a message queue module, a real-time statistical service program module, a database module and a cache module, wherein the real-time statistical service program module obtains the device reported data and the corresponding collection time from the message queue module, and compares and judges the time period with the reported data and the corresponding collection time in the cache, updates the data statistics according to the database statistical rules of the judgment processing result, and updates the latest reported data and the corresponding collection time in the cache module.
[0052] For example, in this specific embodiment, the message queue module may include but is not limited to the statistical group of the Kafka message queue. The device data collected by the IoT device will be uniformly reported to the statistical group of the Kafka message queue. The data in the message queue increases in time for each device. The data structure of the original device data stored in the Kafka message queue is as follows: Figure 3As shown, the data can be in JSON format or in a custom format, and at least include three fields: device unique number, data collection time (time stamp), and table code value (table code data); the cache module can include but is not limited to Redis, which is used to store cached data. The data structure of the data in the Redis cache is as follows Figure 4 shown.
[0053] Specifically, the real-time data statistics method can count the real-time data of hours, days, months and years. The fields of the hourly statistics table are the device unique number, date, hour, start table code, end table code, start table code time, and end table code time. The device unique number + date + hour is the unique key, such as Figure 5A As shown in the figure, the fields of the daily statistics table are the device unique number, date, start table code, end table code, start table code time, and end table code time, where the device unique number + date is the unique key, such as Figure 5B As shown in the figure, the fields of the monthly statistics table are the device unique number, date, start table code, end table code, start table code time, and end table code time. The day in the date is fixed as 1, and the device unique number + date is the unique key. Figure 5C As shown in the figure, the annual statistics table fields are the equipment unique number, year, starting table code, ending table code, starting table code time, and ending table code time, where the equipment unique number + year is the unique key, such as Figure 5D As shown; based on the above design, you can efficiently and conveniently query the statistical data of the equipment at any time of the hour, day, month, or year.
[0054] The embodiment of the present invention also discloses a real-time data statistics device, such as Figure 6 As shown, the device includes: a first acquisition module 201: used to acquire the device reported data and the corresponding collection time pre-stored in the message queue, and form at least the first reported data and the second reported data according to the reported data and the corresponding collection time; a second acquisition module 202: used to acquire the second reported data and the corresponding collection time from the message queue and acquire the first reported data and the corresponding collection time from the buffer area; a comparison module 203: used to compare the collection time corresponding to the first reported data with the collection time corresponding to the second reported data; a first storage module 204: used to store the second reported data in the first storage statistical period of the database when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, and write the second reported data and the corresponding collection time into the buffer area; a second storage module 205: used to store the device data in the second storage statistical period of the database when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, and write the second reported data and the corresponding collection time into the buffer area.
[0055] The real-time data statistics device provided by the present invention comprises: a first acquisition module: used for acquiring device reported data and corresponding collection time pre-stored in a message queue, and forming at least first reported data and second reported data according to the reported data and the corresponding collection time; a second acquisition module: used for acquiring the second reported data and the corresponding collection time from the message queue and acquiring the first reported data and the corresponding collection time from a buffer area; a comparison module: used for comparing the collection time corresponding to the first reported data with the collection time corresponding to the second reported data; a first storage module: used for storing the second reported data in a first storage statistical period of a database when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, and writing the second reported data and the corresponding collection time into a buffer area; a second storage module: used for storing the device data in a second storage statistical period of a database when the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, and writing the second reported data and the corresponding collection time into a buffer area. The device of the present invention is a streaming real-time data statistics device based on time series. Once a table code data reported by a device is obtained, a piece of data is processed in real time. The power consumption of the current statistical period can be counted in real time. At the same time, the real-time data statistics task does not need to count all the data in a statistical period, which effectively reduces the load of the statistical system.
[0056] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, the electronic device may include a processor 401 and a memory 402, wherein the processor 401 and the memory 402 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.
[0057] The processor 401 may be a central processing unit (CPU). The processor 401 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.
[0058] The memory 402 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the real-time data statistics method in the embodiment of the present invention. The processor 401 executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory 402, that is, the real-time data statistics method in the above method embodiment is implemented.
[0059] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 401, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the processor 401 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0060] The one or more modules are stored in the memory 402, and when executed by the processor 401, the following is performed: Figure 1 The real-time data statistics method in the illustrated embodiment.
[0061] For details of the above electronic equipment, please refer to Figure 1 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.
[0062] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.
[0063] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A real-time data statistics method, characterized in that: The steps include: Acquire the device reported data and the corresponding collection time pre-stored in the message queue, and form at least first reported data and second reported data according to the reported data and the corresponding collection time; Acquire the second reported data and the corresponding collection time from the message queue and acquire the first reported data and the corresponding collection time from the buffer area; Comparing the collection time corresponding to the first reported data with the collection time corresponding to the second reported data; When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, the second reported data is stored in the first statistical storage period of the database, and the second reported data and the corresponding collection time are written into the cache area; When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the second reported data in the second storage statistical period of the database, and writing the second reported data and the corresponding collection time into the cache area; The stored statistical period data includes: device unique number, date, statistical period, starting value, ending value, starting value collection time, and ending value collection time; When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data belong to the same statistical period, storing the second reported data in the first storage statistical period of the database includes: The last value of the first reported data in the current statistical period is replaced by the last value of the second reported data stored in the first statistical storage period, the starting value remains unchanged, and the second reported data and the corresponding collection time are written into the cache area to update the data in the cache area; When the collection time corresponding to the first reported data and the collection time corresponding to the second reported data do not belong to the same statistical period, storing the second reported data in the second storage statistical period of the database includes: The first reported data obtained and stored in the end value of the first storage statistical period is used as the starting value of the second storage statistical period, and the second reported data obtained and stored in the end value of the second storage statistical period is used as the end value of the second storage statistical period. The starting value and the end value of the first storage statistical period remain unchanged, and the second reported data and the corresponding collection time are written into the cache area to update the cache area data.
2. The method according to claim 1, characterized in that Before obtaining the device reported data and the corresponding collection time pre-stored in the message queue, and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time, the method further includes: The first reported data and the corresponding collection time are obtained from the message queue for the first time, at least one first storage statistics period is formed in the database, and the first reported data and the collection time obtained for the first time are written into the cache area.
3. The method according to claim 1, characterized in that The types of storage statistics time periods include: storage statistics hourly time period, storage statistics daily time period, storage statistics weekly time period, storage statistics monthly time period, and storage statistics annual time period.
4. The method according to claim 3, characterized in that Before obtaining the device reported data and the corresponding collection time pre-stored in the message queue, and forming at least the first reported data and the second reported data according to the reported data and the corresponding collection time, the method further includes: The first reported data and the corresponding collection time are obtained from the message queue for the first time, and at least two types of storage statistics hourly period, storage statistics daily period, storage statistics weekly period, storage statistics monthly period, and storage statistics annual period are formed in the database, and corresponding first storage statistics periods are formed in different types of storage statistics period, and the first reported data and the collection time obtained for the first time are written into the cache area.
5. The method according to claim 1, characterized in that The device reported data includes: device unique number and table code value.
6. The method according to claim 1, characterized in that The reported data includes: any one of a water metering device and water consumption, a gas metering device and gas consumption, and an electricity metering device and electricity consumption.
7. A real-time data statistics system, executing the real-time data statistics method according to any one of claims 1 to 6, the system comprising: A message queue module, a real-time statistical service program module, a database module and a cache module, wherein the real-time statistical service program module obtains the device reported data and the corresponding collection time from the message queue module, and compares and judges the time period with the reported data and the corresponding collection time in the cache, updates the data statistics according to the database statistical rules, and updates the latest reported data and the corresponding collection time in the cache module.
Citation Information
Patent Citations
Monitoring data processing method and device
CN105868071A
