Online and offline message processing method, system, equipment and medium

Abstract

The invention provides an online and offline message processing method and system, equipment and a medium, and the method comprises the steps: obtaining a current time window, a current table index, a first online and offline message of the equipment and a first data table of a database; the time length of the current time window is larger than the time length of the previous time window when the first online and offline messages are stacked, and the number of the joint primary key elements in the current table index is smaller than or equal to the number of the joint primary key elements in the previous table index when the first online and offline messages are stacked; according to the current time window, performing first message aggregation on the first online and offline message to obtain first equipment aggregation data; and according to the current table index and the first equipment aggregation data, performing first storage updating on the first data table to obtain an equipment online and offline processing result. The method can effectively improve the consumption rate of the online and offline messages, inhibit the accumulation of the online and offline messages, and reduce the processing delay of the online and offline messages. The invention relates to the technical field of data processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, system, device and medium for processing online and offline messages. Background Art

[0002] The two key elements of the Internet of Things are IoT devices and communication networks. When an IoT device fails, it is easy for the IoT device to frequently go online and offline; and when the communication network fails, it is easy for the IoT devices connected to the communication network to frequently go online and offline or go offline for a long time. Therefore, the IoT platform can monitor and count the number of times IoT devices go online and offline as one of the pre-analysis methods for fault diagnosis.

[0003] The relevant technology usually collects the online and offline message data generated by each IoT device within a preset time range, and then further aggregates the data of all the online and offline messages read. When the number of IoT devices is large (for example, tens of millions, hundreds of millions, billions, etc.), the number of online and offline messages collected is also large, and the computing resources required for data aggregation are relatively large, which may easily lead to serious accumulation of online and offline messages and high latency.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] The main purpose of the embodiments of the present application is to propose a method for processing online and offline messages, which can effectively increase the consumption rate of online and offline messages, suppress the accumulation of online and offline messages, and reduce the delay in processing online and offline messages.

[0007] To achieve the above purpose, an embodiment of the present application provides a method for processing online and offline messages, the method comprising:

[0008] Acquire a current time window and a current table index, as well as a first online and offline message of a device and a first data table of a database, wherein the time length of the current time window is greater than the time length of a previous time window when the first online and offline messages are accumulated, and the number of joint primary key elements in the current table index is less than or equal to the number of joint primary key elements in the previous table index when the first online and offline messages are accumulated;

[0009] According to the current time window, performing first message aggregation on the first online and offline messages to obtain first device aggregate data, where the first device aggregate data is used to indicate the number of online and offline times of the device within the current time window;

[0010] According to the current table index and the first device aggregated data, a first storage update is performed on the first data table to obtain a device online and offline processing result.

[0011] In some embodiments, performing first message aggregation on the first online and offline messages according to the current time window to obtain first device aggregate data includes:

[0012] According to the current time window, the first online and offline messages are screened to obtain first window messages corresponding to the current time window;

[0013] The first window messages are grouped and aggregated according to a preset grouping type to obtain the first device aggregated data.

[0014] In some embodiments, performing a first storage update on the first data table according to the current table index and the first device aggregated data to obtain a device online and offline processing result includes:

[0015] According to the current table index, verify the existence of a primary key of the first data table to obtain a first primary key verification result, where the first primary key verification result is used to indicate whether the first data table has a primary key element corresponding to the current table index;

[0016] If the first primary key verification result is existence, the first data table is supplemented and updated according to the first device aggregate data to obtain a second data table, and the device online and offline processing result is generated based on the second data table; or, if the first primary key verification result is non-existence, the first data table is inserted and updated according to the first device aggregate data to obtain a third data table, and the device online and offline processing result is generated based on the third data table.

[0017] In some embodiments, performing a first storage update on the first data table according to the current table index and the first device aggregated data to obtain a device online and offline processing result includes:

[0018] a second online and offline message is obtained according to the aggregated data of the first device, wherein the message queue to which the second online and offline message belongs is different from the message queue to which the first online and offline message belongs;

[0019] According to a preset target window, performing second message aggregation on the second online and offline messages to obtain second device aggregated data;

[0020] According to the current table index and the second device aggregated data, a second storage update is performed on the first data table to obtain the device online and offline processing result.

[0021] In some embodiments, if the target window is a target time window, performing second message aggregation on the second online and offline messages according to the preset target window to obtain second device aggregate data includes:

[0022] According to the target time window, the second online and offline messages are screened to obtain second window messages corresponding to the target time window;

[0023] The second window messages are grouped and aggregated according to a preset grouping type to obtain the second device aggregated data.

[0024] In some embodiments, if the target window is a target counting window, performing second message aggregation on the second online and offline messages according to the preset target window to obtain second device aggregate data includes:

[0025] According to the target counting window, the second online and offline messages are screened to obtain a third window message corresponding to the target counting window;

[0026] The third window message is grouped and aggregated according to a preset grouping type to obtain the second device aggregated data.

[0027] In some embodiments, performing a second storage update on the first data table according to the current table index and the second device aggregated data to obtain the device online and offline processing result includes:

[0028] Obtaining a historical data table of the database;

[0029] According to a preset expiration condition, the first data table is screened to obtain a fourth data table;

[0030] According to the current table index, verify the existence of the primary key of the fourth data table to obtain a second primary key verification result, where the second primary key verification result is used to indicate whether the fourth data table has a primary key element corresponding to the current table index;

[0031] If the second primary key verification result is existence, the fourth data table is supplemented and updated according to the second device aggregate data to obtain the fifth data table, and the device online and offline processing result is generated based on the fifth data table and the historical data table; or, if the second primary key verification result is non-existence, the fourth data table is inserted and updated according to the second device aggregate data to obtain the sixth data table, and the device online and offline processing result is generated based on the sixth data table and the historical data table.

[0032] To achieve the above purpose, another aspect of the embodiment of the present application provides an online and offline message processing system, including:

[0033] an acquisition unit, used to acquire a current time window and a current table index, as well as a first online and offline message of a device and a first data table of a database, wherein the time length of the current time window is greater than the time length of a previous time window when the first online and offline messages are accumulated, and the number of joint primary key elements in the current table index is less than or equal to the number of joint primary key elements in the previous table index when the first online and offline messages are accumulated;

[0034] an aggregation unit, configured to perform first message aggregation on the first online and offline messages according to the current time window to obtain first device aggregate data, wherein the first device aggregate data is used to indicate the number of online and offline times of the device within the current time window;

[0035] A processing unit is used to perform a first storage update on the first data table according to the current table index and the first device aggregated data to obtain a device online and offline processing result.

[0036] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0037] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0038] The embodiments of the present application include at least the following beneficial effects:

[0039] The present application provides an online and offline message processing method, system, device and medium, wherein the processing method obtains a current time window and a current table index, as well as a first online and offline message of a device and a first data table of a database, wherein the time length of the current time window is greater than the time length of a previous time window when the first online and offline messages are accumulated, and the number of joint primary key elements in the current table index is less than or equal to the number of joint primary key elements in the previous table index when the first online and offline messages are accumulated; according to the current time window, the first online and offline messages are subjected to a first message aggregation to obtain first device aggregation data, and the first device aggregation data is used to indicate the number of online and offline times of the device within the current time window; according to the current table index and the first device aggregation data, the first data table is subjected to a first storage update to obtain a device online and offline processing result. When the first online and offline messages are accumulated, the processing method aggregates the first online and offline messages based on the current time window with a larger time length, which can effectively improve the consumption rate of the first online and offline messages, suppress the accumulation of the first online and offline messages, and reduce the delay; in addition, the processing method also updates the first data table based on the current table index and the first device aggregated data, which can further improve the consumption rate of the first online and offline messages, and can also suppress the accumulation of the first online and offline messages and reduce the delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of a method for processing online and offline messages provided in an embodiment of the present application;

[0041] Figure 2 It is a framework flow chart of a method for processing online and offline messages provided in an embodiment of the present application;

[0042] Figure 3 is a detailed flow chart of step S120 provided in an embodiment of the present application;

[0043] Figure 4 is a detailed flow chart of the first step S130 provided in an embodiment of the present application;

[0044] Figure 5 is a detailed flow chart of the second step S130 provided in an embodiment of the present application;

[0045] Figure 6 is a detailed flow chart of the first step C2 provided in the embodiment of the present application;

[0046] Figure 7 is a detailed flow chart of the second step C2 provided in the embodiment of the present application;

[0047] Figure 8 is a detailed flow chart of step C3 provided in an embodiment of the present application;

[0048] Fig. 9 It is a schematic diagram of the framework of an online and offline message processing system provided in an embodiment of the present application;

[0049] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices / equipment and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] The two key elements of the Internet of Things are IoT devices and communication networks. When an IoT device fails, such as power system failure, module integration failure, device certificate problem, device-side software bug, IoT platform instability, etc., it is easy for the IoT device to frequently go online and offline; and when the communication network fails, such as physical environment, communication load, bandwidth limitation, hardware failure and other failure factors, it is easy for IoT devices connected to the communication network to frequently go online and offline or be offline for a long time. Therefore, monitoring and counting the number of times IoT devices go online and offline can be used as one of the pre-analysis methods for fault diagnosis.

[0055] The relevant technology usually collects the online and offline message data generated by each IoT device within a preset time range, and then further aggregates the data of all the online and offline messages read. When the number of IoT devices is large (for example, tens of millions, hundreds of millions, billions, etc.), the number of online and offline messages collected is also large, and the computing resources required for data aggregation are relatively large, which may easily lead to serious accumulation of online and offline messages and high latency.

[0056] In view of this, the present application provides an online and offline message processing method, system, device and medium, wherein the processing method, when the first online and offline messages are accumulated, aggregates the first online and offline messages based on the current time window with a larger time length, which can effectively improve the consumption rate of the first online and offline messages, suppress the accumulation of the first online and offline messages, and reduce delays; in addition, the processing method also updates the first data table based on the current table index and the first device aggregated data, which can further improve the consumption rate of the first online and offline messages, and can also suppress the accumulation of the first online and offline messages and reduce delays.

[0057] The embodiment of the present application provides a method for processing online and offline messages, which can be applied to the application scenario of the Internet of Things. In the application scenario of the Internet of Things, the Internet of Things service provider can process the online and offline messages sent by each Internet of Things device in the Internet of Things through the online and offline message processing method provided by the embodiment of the present application, and can suppress the occurrence of a large number of online and offline message accumulation phenomena when the order of magnitude of the online and offline messages is large, and the consumption rate of the online and offline messages is high, and the processing delay is low.

[0058] The embodiment of the present application provides a method for processing online and offline messages, which can be applied to a terminal or a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method, etc., but is not limited to the above forms.

[0059] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0060] Reference Figure 1 and Figure 2 , Figure 1 This is an optional flow chart of a method for processing online and offline messages provided in an embodiment of the present application. Figure 1 The processing method may include but is not limited to steps S110 to S130.

[0061] Step S110: acquiring a current time window and a current table index, as well as a first online and offline message of a device and a first data table of a database, wherein the time length of the current time window is greater than the time length of a previous time window when the first online and offline messages are accumulated, and the number of joint primary key elements in the current table index is less than or equal to the number of joint primary key elements in the previous table index when the first online and offline messages are accumulated;

[0062] In an embodiment of the present application, the time window is used to convert an unbounded online and offline message data stream into a bounded online and offline message data stream in chronological order, which can specifically be a TimeWindow in the Flink distributed stream processing framework; the table index is used to indicate the data in the first data table and the corresponding storage location.

[0063] It can be understood that the first online and offline message specifically includes fields such as tenant ID, product ID, device ID, event type, and timestamp, wherein when the event type is 0, it indicates that the IoT device is offline, and when the event type is 1, it indicates that the IoT device is online. Exemplarily, for an IoT device, it can send an online and offline message to the Pu l sar distributed message platform when it goes online or offline. After receiving several online and offline messages of the IoT device, the Pu l sar distributed message platform can create a message queue for processing according to the several online and offline messages, and the message queue is recorded as the first message queue. The first data table is used to store the online and offline messages after data aggregation, which can specifically be a data table in the Ti DB database or a data table in the HBASE database.

[0064] It should be noted that the judgment of the accumulation of the first online and offline messages can be based on one or more preset accumulation judgment thresholds, compared with the total number of the first online and offline messages in the first message queue, and then whether the first online and offline messages are accumulated can be concluded based on the comparison result; or, it can be based on the time series, by analyzing the changing trend of the number of first online and offline messages in the first message queue. For example, within a certain sampling time period, the changing trend of the number of first online and offline messages in the first message queue is an increasing trend, then it can be considered that within the sampling time period, the first online and offline messages are in an accumulated state, and the changing trend of the number of first online and offline messages in the first message queue is a decreasing trend, then it can be considered that within the sampling time period, the first online and offline messages are in a non-accumulated state. There are many ways to determine whether the first online and offline messages are accumulated, which will not be elaborated in this application.

[0065] It is worth mentioning that when the first online and offline messages are accumulated, the change range of the time length between the current time window and the previous time window can be set according to the actual situation. Specifically, one or more specific length change thresholds can be set based on the specific number of the first online and offline messages accumulated, and then the corresponding length change threshold is selected according to the specific number of the first online and offline messages accumulated, and the time length of the previous time window, and the time length of the current time window is calculated; or, by performing data analysis on historical data, a relationship curve diagram between the time length of the time window and the number of accumulated first online and offline messages can be obtained, and the time length of the current time window can be determined through the relationship curve diagram. The examples in this application are for illustration only and are not intended to limit this application. In addition, the number of joint primary key elements in the current table index is similar to the time length of the current time window mentioned above, and can be simply deduced by analogy, and this application will not elaborate on it here.

[0066] Step S120: performing first message aggregation on the first online and offline messages according to the current time window to obtain first device aggregate data, where the first device aggregate data is used to indicate the number of online and offline times of the device in the current time window;

[0067] Reference Figure 3 In some embodiments, the step S120, performing first message aggregation on the first online and offline messages according to the current time window to obtain first device aggregate data, includes:

[0068] A1. Filter the first online and offline messages according to the current time window to obtain a first window message corresponding to the current time window;

[0069] A2. Group and aggregate the first window messages according to a preset grouping type to obtain the first device aggregate data.

[0070] In an embodiment of the present application, step A1 can perform real-time processing of the first online and offline messages from different devices in the form of a DataStream data stream in the Flink framework according to the current time window. Specifically, all the first online and offline messages in the TimeWi ndow corresponding to the current moment can be selected as the first window message in step A1.

[0071] It can be understood that the grouping type can be any one of the field types such as tenant id (tenant I d), product id (product I d), device id (device I d). Specifically, step A2 can be to group all the first online and offline messages in the first window message according to any one of the tenant id, product id, device id, etc., to obtain a number of grouped first window messages; then, for the first window message of a certain grouping type, the number of online and offline times of the device under the grouping type in the current time window can be obtained by statistics, and the number of online and offline times of the device obtained by statistics in the current time window is used as the first device aggregate data. It should be noted that in the embodiment of the present application, the grouping type can also be a combination of field types such as tenant id (tenant I d), product id (product I d), device id (dev i ce I d), etc., which will not be repeated in this application.

[0072] Step S130: Perform a first storage update on the first data table according to the current table index and the first device aggregated data to obtain a device online and offline processing result.

[0073] Reference Figure 4 In some embodiments, the step S130, performing a first storage update on the first data table according to the current table index and the first device aggregated data, and obtaining a device online and offline processing result, includes:

[0074] B1. According to the current table index, verify the existence of the primary key of the first data table to obtain a first primary key verification result, where the first primary key verification result is used to indicate whether the first data table has a primary key element corresponding to the current table index;

[0075] B2. If the first primary key verification result is existence, the first data table is supplemented and updated according to the first device aggregated data to obtain a second data table, and the device online and offline processing result is generated based on the second data table;

[0076] Alternatively, B3, if the first primary key verification result is non-existent, the first data table is inserted and updated according to the first device aggregation data to obtain a third data table, and the device online and offline processing result is generated based on the third data table.

[0077] In an embodiment of the present application, the first inventory update in step S130 is used to write the obtained first device aggregate data into the database. Specifically, it can be based on the DataSi nk Function component in the Flink framework to write the number of online and offline times of each device in the current time window into the TiDB database.

[0078] It is understandable that step B1 can search the first data table in the database based on the joint primary key element contained in the table index corresponding to the current moment, so as to obtain the first primary key verification result. Specifically, if the first data table contains information corresponding to the joint primary key element, a first verification result indicating that the first data table has the primary key element corresponding to the current table index can be obtained; or, if the first data table contains information corresponding to the joint primary key element, a first verification result indicating that the first data table does not have the primary key element corresponding to the current table index can be obtained.

[0079] It should be noted that if the first primary key verification result is existence, the supplementary update in step B2 can be to add the first device aggregate data to the position indicated by the corresponding primary key element in the first data table; or, if the first primary key verification result is not existence, the insertion update in step B3 can be to create a corresponding primary key element in the first data table, and then add the first device aggregate data to the position indicated by the newly inserted primary key element.

[0080] It is worth mentioning that the online and offline processing results of the production equipment in the second data table or the third data table can be simply obtained by counting the online and offline messages recorded by each IoT device in the second data table or the third data table, which will not be elaborated in this application.

[0081] It should be added that, in actual applications, data aggregation based on the Flink framework is usually performed in memory. When the first online and offline messages accumulate, the embodiment of the present application aggregates the first online and offline messages based on the current time window with a larger time length, so that the load brought by the first context message of the Internet of Things device can be transferred to the memory for processing, which can reduce the frequency of database operations, improve the consumption efficiency of the first online and offline messages, suppress the accumulation of the first online and offline messages, and reduce latency. In addition, the embodiment of the present application is also based on the current table index that is less than or equal to the number of primary key elements combined with the previous table index, and further improves the consumption rate of the first online and offline messages by reducing the performance overhead required for updating the data table, and can also suppress the accumulation of the first online and offline messages and reduce latency.

[0082] It is worth noting that when online and offline messages accumulate, increasing the length of the time window an unlimited number of times can effectively reduce the consumption rate of the first online and offline messages in the first message queue, but it may reduce the real-time performance of database data query.

[0083] Reference Figure 5 In some other embodiments, the step S130, performing a first storage update on the first data table according to the current table index and the first device aggregated data, and obtaining a device online and offline processing result, includes:

[0084] C1. Obtain a second online and offline message according to the aggregated data of the first device, wherein the message queue to which the second online and offline message belongs is different from the message queue to which the first online and offline message belongs;

[0085] In an embodiment of the present application, the second online and offline message can be obtained based on the aggregated data of the first device. Specifically, a new message queue can be created in the Flink framework, and the new message queue is recorded as the second message queue. The second message queue is used to receive the aggregated data of the first device, and the received aggregated data of the first device is used as the second online and offline message. The second message queue and the first message queue have an isolation relationship in the Flink framework, that is, the message queue to which the second online and offline message belongs is different from the message queue to which the first online and offline message belongs.

[0086] C2. Performing second message aggregation on the second online and offline messages according to a preset target window to obtain second device aggregated data;

[0087] Reference Figure 6 Further, if the target window is a target time window, the step C2, according to the preset target window, performs second message aggregation on the second online and offline messages to obtain second device aggregate data, includes:

[0088] C21. Filter the second online and offline messages according to the target time window to obtain second window messages corresponding to the target time window;

[0089] C22. Group and aggregate the second window messages according to a preset grouping type to obtain the second device aggregate data.

[0090] In the embodiment of the present application, the target time window can be a time window of a preset fixed time length, or a time window of a dynamically changing time length. Specifically, if the target time window is a time window of a dynamically changing time length, the time length of the target time window can be positively correlated based on the specific number of the second online and offline messages in the second message queue. The example of the present application is only for illustration. It can be understood that for step C21 and step C22, the contents of the other aforementioned steps A1 and A2 are similar and can be simply deduced by analogy, and the present application will not elaborate on them here.

[0091] C3. Perform a second storage update on the first data table according to the current table index and the second device aggregated data to obtain the device online and offline processing result.

[0092] In the embodiment of the present application, the second inventory update in step C3 is similar to the first inventory update in step S130 and can be simply deduced by analogy, so this application will not elaborate on it here.

[0093] It should be added that, compared with the aforementioned method of directly obtaining the device online and offline processing result based on the aggregated data of the first device, the embodiment of the present application receives the aggregated data of the first device based on the second message queue and generates the second online and offline message, which can further improve the consumption rate of the first online and offline messages in the first message queue, suppress the accumulation of online and offline messages in the first message queue, and reduce the delay in online and offline message processing; in addition, the embodiment of the present application obtains the device online and offline processing result by performing secondary aggregation on the second online and offline messages in the second message queue and then writing them into the database, which can further reduce the frequency of database operations and improve the real-time performance of database data queries.

[0094] It is worth mentioning that in actual applications, due to the large number of IoT devices, the specific number of which is often in the tens of millions, hundreds of millions, billions, etc., the database (such as the TiDB database) will add tens of millions or hundreds of millions of device online and offline records every day. The execution speed of the duplicate key value check operation in the database will decrease with the increase of device online and offline records in the database, thereby reducing the consumption rate of the second online and offline messages in the second message queue, and thereby reducing the consumption rate of the first online and offline messages; in addition, due to the uneven arrival rate of the aggregated data of the first device, when the second online and offline messages are screened and grouped, there are situations where the amount of data in some target time windows is large, while the amount of data in other target time windows is small, resulting in fluctuations in the consumption rate of the second online and offline messages in the second message queue, and thereby reducing the consumption rate of the first online and offline messages in certain time periods.

[0095] Reference Figure 7 In some other embodiments, if the target window is a target counting window, the step C2, performing second message aggregation on the second online and offline messages according to the preset target window to obtain second device aggregate data, includes:

[0096] C23. Filter the second online and offline messages according to the target counting window to obtain a third window message corresponding to the target counting window.

[0097] C24. Group and aggregate the third window messages according to a preset grouping type to obtain the second device aggregated data.

[0098] In the embodiment of the present application, the target counting window is used to select a fixed number of second online and offline messages to obtain a plurality of third window messages, and the number of second context messages between each third window message is the same. Step C24 is similar to the aforementioned step A2, and can be simply deduced by analogy, and the present application will not elaborate on it here.

[0099] It can be understood that the embodiment of the present application can obtain the same number of third window messages by filtering the second online and offline messages through the target counting window, which can reduce the fluctuation range of the second online and offline message consumption rate of the second message queue, thereby improving the overall consumption rate of the first online and offline messages.

[0100] Reference Figure 8 The step C3, performing a second storage update on the first data table according to the current table index and the second device aggregated data, and obtaining the device online and offline processing result, includes:

[0101] C31, obtaining a historical data table of the database;

[0102] C32. Screen the first data table according to a preset expiration condition to obtain a fourth data table;

[0103] C33. According to the current table index, verify the existence of the primary key of the fourth data table to obtain a second primary key verification result, where the second primary key verification result is used to indicate whether the fourth data table has a primary key element corresponding to the current table index;

[0104] C34. If the second primary key verification result is existence, the fourth data table is supplemented and updated according to the second device aggregated data to obtain a fifth data table, and the device online and offline processing result is generated based on the fifth data table and the historical data table;

[0105] Alternatively, C35, if the second primary key verification result is non-existent, insert and update the fourth data table according to the second device aggregate data to obtain the sixth data table, and generate the device online and offline processing result based on the sixth data table and the historical data table.

[0106] In an embodiment of the present application, the first data table of the database and the historical data table of the database can specifically be two data tables in the TiDB database, wherein the first data table and the historical data table are independent of each other and do not affect each other, and the historical data table is used to store context messages of IoT devices in historical time periods.

[0107] It is understandable that the expiration condition is used to determine whether the aggregated context information in the first data table is expired, and the expiration condition can specifically be a date time period. Specifically, the expiration condition can be set based on the TTL (Time To Live) mechanism in the TiDB database, and the data cleaning of the first data table is implemented according to the creation time corresponding to the aggregated context information of the first data table. For example, when the expiration condition can be to retain only the data of the current day, the historical data table can be used to store the context messages of the IoT device before the current day, and the fourth data table obtained in step C32 only retains the aggregated context information with the creation time of the current day.

[0108] It should be noted that step C33 is similar to the content of the aforementioned step B1 and can be simply deduced by analogy. In the embodiment of the present application, the expiration condition is to retain only the data of the current day as an example. The generation of the device online and offline processing result based on the fifth data table and the historical data table in step C34 can be simply obtained by counting the context messages of the historical data table before the current day and the context of the fifth data table on the current day. The rest of the content of step C34 and step C35 is similar to the content of the aforementioned step B2 and can be simply deduced by analogy.

[0109] It is worth noting that the first data table in the embodiment of the present application can specifically be a data table in the TiDB database, and the historical data table can be a data table in the HBASE database. The context information in the first data table is periodically synchronized to the historical data table in the HBASE database through the Canal data synchronization tool.

[0110] It is worth mentioning that the embodiment of the present application performs data filtering on the first data table based on the expiration condition, which can reduce the context information recorded in the first data table, thereby improving the execution speed of the duplicate key value check operation in the database, and further increasing the consumption rate of the second context message in the second message queue, and the consumption rate of the first context message in the first message queue.

[0111] See also Fig. 9 The embodiment of the present application also provides an online and offline message processing system, the processing system comprising:

[0112] The acquisition unit 810 is used to acquire a current time window and a current table index, as well as a first online and offline message of a device and a first data table of a database, wherein the time length of the current time window is greater than the time length of a previous time window when the first online and offline messages are accumulated, and the number of joint primary key elements in the current table index is less than or equal to the number of joint primary key elements in the previous table index when the first online and offline messages are accumulated;

[0113] An aggregation unit 820 is used to perform first message aggregation on the first online and offline messages according to the current time window to obtain first device aggregate data, where the first device aggregate data is used to indicate the number of online and offline times of the device in the current time window;

[0114] The processing unit 830 is used to perform a first storage update on the first data table according to the current table index and the first device aggregated data to obtain a device online and offline processing result.

[0115] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0117] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0118] See also Fig.10 , Fig.10 The hardware structure of an electronic device of an embodiment is illustrated, and the electronic device includes:

[0119] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0120] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 calls and executes the methods of the embodiments of this application;

[0121] Input / output interface 903, used to implement information input and output;

[0122] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0123] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0124] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0125] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0126] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0127] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0129] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0130] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0132] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0133] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0138] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for processing online and offline messages, characterized in that: include: Acquire a current time window and a current table index, as well as a first online and offline message of a device and a first data table of a database, wherein the time length of the current time window is greater than the time length of a previous time window when the first online and offline messages are accumulated, and the number of joint primary key elements in the current table index is less than or equal to the number of joint primary key elements in the previous table index when the first online and offline messages are accumulated; According to the current time window, performing first message aggregation on the first online and offline messages to obtain first device aggregate data, where the first device aggregate data is used to indicate the number of online and offline times of the device within the current time window; According to the current table index and the first device aggregated data, a first storage update is performed on the first data table to obtain a device online and offline processing result.

2. The processing method according to claim 1, characterized in that: The performing first message aggregation on the first online and offline messages according to the current time window to obtain first device aggregated data includes: According to the current time window, the first online and offline messages are screened to obtain first window messages corresponding to the current time window; The first window messages are grouped and aggregated according to a preset grouping type to obtain the first device aggregated data.

3. The processing method according to claim 1, characterized in that: The first data table is updated according to the current table index and the first device aggregated data to obtain the device online and offline processing result, including: According to the current table index, verify the existence of a primary key of the first data table to obtain a first primary key verification result, where the first primary key verification result is used to indicate whether the first data table has a primary key element corresponding to the current table index; If the first primary key verification result is existence, the first data table is supplemented and updated according to the first device aggregate data to obtain a second data table, and the device online and offline processing result is generated based on the second data table; or, if the first primary key verification result is non-existence, the first data table is inserted and updated according to the first device aggregate data to obtain a third data table, and the device online and offline processing result is generated based on the third data table.

4. The processing method according to claim 1, characterized in that: The first data table is updated according to the current table index and the first device aggregated data to obtain the device online and offline processing result, including: a second online and offline message is obtained according to the aggregated data of the first device, wherein the message queue to which the second online and offline message belongs is different from the message queue to which the first online and offline message belongs; According to a preset target window, performing second message aggregation on the second online and offline messages to obtain second device aggregated data; According to the current table index and the second device aggregated data, a second storage update is performed on the first data table to obtain the device online and offline processing result.

5. The processing method according to claim 4, characterized in that: If the target window is a target time window, performing second message aggregation on the second online and offline messages according to the preset target window to obtain second device aggregated data includes: According to the target time window, the second online and offline messages are screened to obtain second window messages corresponding to the target time window; The second window messages are grouped and aggregated according to a preset grouping type to obtain the second device aggregated data.

6. The processing method according to claim 4, characterized in that: If the target window is a target counting window, performing second message aggregation on the second online and offline messages according to the preset target window to obtain second device aggregated data includes: According to the target counting window, the second online and offline messages are screened to obtain a third window message corresponding to the target counting window; The third window message is grouped and aggregated according to a preset grouping type to obtain the second device aggregated data.

7. The processing method according to claim 6, characterized in that: The performing a second storage update on the first data table according to the current table index and the second device aggregated data to obtain the device online and offline processing result includes: Obtaining a historical data table of the database; According to a preset expiration condition, the first data table is screened to obtain a fourth data table; According to the current table index, verify the existence of the primary key of the fourth data table to obtain a second primary key verification result, where the second primary key verification result is used to indicate whether the fourth data table has a primary key element corresponding to the current table index; If the second primary key verification result is existence, the fourth data table is supplemented and updated according to the second device aggregate data to obtain the fifth data table, and the device online and offline processing result is generated based on the fifth data table and the historical data table; or, if the second primary key verification result is non-existence, the fourth data table is inserted and updated according to the second device aggregate data to obtain the sixth data table, and the device online and offline processing result is generated based on the sixth data table and the historical data table.

8. A system for processing online and offline messages, characterized in that: include: an acquisition unit, used to acquire a current time window and a current table index, as well as a first online and offline message of a device and a first data table of a database, wherein the time length of the current time window is greater than the time length of a previous time window when the first online and offline messages are accumulated, and the number of joint primary key elements in the current table index is less than or equal to the number of joint primary key elements in the previous table index when the first online and offline messages are accumulated; an aggregation unit, configured to perform first message aggregation on the first online and offline messages according to the current time window to obtain first device aggregate data, wherein the first device aggregate data is used to indicate the number of online and offline times of the device within the current time window; A processing unit is used to perform a first storage update on the first data table according to the current table index and the first device aggregated data to obtain a device online and offline processing result.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images