Method, system, electronic device and storage medium for real-time processing of data

Abstract

The application discloses a kind of for real-time processing data method, applied to the system of real-time processing data, the system includes receiving module, judging module and processing module, method includes: the receiving module receives data from equipment end, wherein the equipment end is used to complete the production of commodity in order;Judging module judges whether the data is transaction information data, in the case where the data is transaction information data, the receiving module sends call information to the processing module;The processing module receives the call information, and the transaction information data is processed based on business index, to obtain business index data, and the business index data is used to represent the business index.In the present application, since the step of judging is added, and the processing module is called only in the case where the judgment result is that the received data is transaction information data, therefore the processing module is not always in working condition, so that cost can be saved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method for real-time data processing, a system for real-time data processing, an electronic device, and a storage medium. Background Art

[0002] With the advancement of science and technology, the vast majority of enterprises have a need for real-time data processing based on their business needs. Generally speaking, the real-time data processing architectures currently adopted by most enterprises are based on mature open-source data processing frameworks, such as Spark Streaming and Apache Flink. Spark Streaming is suitable for the rapid processing of large amounts of data. Its core concept is data micro-batching. When the data window for batch processing is extremely small, it can be roughly considered data stream processing, enabling rapid expansion of real-time data and featuring high throughput and high fault tolerance. Apache Flink is also suitable for the rapid processing of large amounts of data and is more mainstream, adopting the concept of integrated streaming and batch processing to truly process streaming data. Flink's greatest feature is its guaranteed correctness in real-time calculations, native support for event time, and support for delayed data processing. Because Flink itself is based on native data stream calculations, it can achieve millisecond-level low latency.

[0003] Specifically, the current real-time data processing architecture mainly includes a storage unit and a processing unit based on the Apache Flink or Spark Streaming architecture. The storage unit is used to store data uploaded by the device, and the processing unit is used to monitor in real time whether there is new data stored in the storage unit, and is also used to perform real-time processing on the data uploaded to the storage unit.

[0004] Regarding the device side, for example, in the retail sector, the device side can be understood as a device for making beverages, such as an automatic coffee vending machine. The data uploaded to the storage unit by the device side can be divided into two categories: business data and log records. Business data mainly includes transaction information data and device information data. Specifically, transaction information data includes order amount, order details, beverage type, discount information, etc.; device information mainly includes device status, material status, etc. Log records can be understood as some real-time messages, such as when a device goes online, a record is generated, and when a device goes offline, another record is generated.

[0005] Regarding the processing unit, as described above, it needs to continuously monitor the storage unit for new data. If new data is stored in the storage unit, the processing unit will process this data in real time based on business indicators, thereby obtaining business indicator data. Business indicators can be device-specific, such as the number of orders completed by device A on the same day, the number of orders completed by device A at different time periods on the same day (hourly dimension); or the number of orders completed by device A on a weekly basis, the total number of beverages sold, and the total order amount. Alternatively, if there are many devices leased to different tenants, business indicators could also include the real-time order flow and number of orders for different tenants on the same day, the real-time order flow and number of orders for different tenants at different time periods on the same day (hourly dimension); the number of beverages sold and the total amount of each beverage sold by different tenants on the same day in reverse order; the number of sales orders, number of cups sold, and the total amount of each beverage sold by different tenants on a weekly basis; and the number of sales orders, number of cups sold, and the total amount of each beverage sold by different tenants on a monthly basis. Therefore, real-time processing here can be understood as processing the received data based on different business indicators to obtain business indicator data representing the business indicators.

[0006] As can be seen from the above description, the processing units in existing real-time processing architectures must always be in a working state, requiring constant monitoring even when no new data is being stored in the storage unit. This constant need for processing units to be in a working state leads to high costs. Furthermore, while native data processing frameworks like Spark Streaming and Apache Flink are powerful and mature, for enterprises with relatively small data volumes, building a real-time data processing architecture using these mature data frameworks carries high development and maintenance costs.

[0007] Therefore, a new method for real-time data processing is needed to reduce development costs and operation and maintenance costs. Summary of the Invention

[0008] In order to at least partially solve the problems existing in the prior art, according to one aspect of the present invention, the present invention provides a method for real-time data processing.

[0009] The method for real-time data processing described in the present invention is applied to a system for real-time data processing, wherein the system includes a receiving module, a judgment module and a processing module, and the method includes: the receiving module receives data from a device end, wherein the device end is used to complete the production of goods in an order; the judgment module determines whether the data is transaction information data, and if the data is transaction information data, the receiving module sends a call message to the processing module; the processing module receives the call message, processes the transaction information data based on business indicators to obtain business indicator data, and the business indicator data is used to characterize the business indicators.

[0010] As an example, in the method for real-time data processing according to the present invention, the processing module receives the call information and processes the transaction information data based on the business indicator to obtain business indicator data, including:

[0011] The processing module receives the calling information;

[0012] After the processing module receives the call information, the transaction information data is marked according to a predetermined dimension to obtain the transaction information data under the predetermined dimension;

[0013] The processing module processes the transaction information data under the predetermined dimension based on the business indicator to obtain business indicator data.

[0014] As another example, in the method for real-time data processing according to the present invention, the processing module processes the transaction information data under the predetermined dimension based on the business indicator to obtain the business indicator data, including:

[0015] The processing module processes the transaction information data under the predetermined dimension based on the business indicators according to a first-in-first-out mechanism to obtain business indicator data.

[0016] As another example, in the method for real-time data processing according to the present invention, the processing module processes the transaction information data under the predetermined dimension according to a first-in-first-out mechanism based on the business indicator to obtain business indicator data, including:

[0017] The processing module inputs the marked transaction information data under the predetermined dimension into a first-in-first-out queue;

[0018] Based on the business indicators, the transaction information data output from the first-in-first-out queue is processed to obtain business indicator data.

[0019] As another example, in the method for real-time data processing according to the present invention, the transaction information data is processed based on the business indicator to obtain the business indicator data, including:

[0020] Parsing the transaction information data to obtain instantaneous status data, wherein the instantaneous status data includes the latest status data of each order and the latest completion status of each product in the order;

[0021] Aggregating the instantaneous state data to obtain business indicator data;

[0022] The method further comprises:

[0023] The business indicator data and the instantaneous state data are stored in different storage modules.

[0024] As another example, in the method for real-time data processing of the present invention, the storing of the business indicator data and the instantaneous state data in different storage modules includes:

[0025] The business indicator data is stored in a MongoDB database, and the transient state data is stored in a DynamoDB database.

[0026] As another example, in the method for real-time data processing described in the present invention, determining whether the data is transaction information data includes: determining whether the data contains preset flag information, and if so, determining that the data is transaction information data.

[0027] According to another aspect of the present invention, the present invention also provides a system for real-time data processing.

[0028] The system for real-time data processing of the present invention includes a receiving module for receiving data from a device, wherein the device is used to complete the production of goods in an order;

[0029] a judging module, configured to judge whether the data is transaction information data; if the data is transaction information data, the receiving module sends a call message to the processing module;

[0030] A processing module is configured to receive the call information from the receiving module and process the transaction information data based on the business indicator to obtain business indicator data, wherein the business indicator data is used to represent the business indicator.

[0031] As an example, in the system for real-time data processing according to the present invention, the processing module includes:

[0032] a first processing module, configured to receive the call information, and after receiving the call information, mark the transaction information data according to a predetermined dimension to obtain the transaction information data under the predetermined dimension;

[0033] The second processing module is used to process the transaction information data under the predetermined dimension based on the business indicator to obtain business indicator data.

[0034] As another example, the system for real-time data processing described herein includes a message transmission module and a queue control module. The message transmission module is configured to broadcast messages corresponding to transaction information data within a predetermined dimension, and the queue control module is configured to receive the messages broadcast by the message transmission module and transmit them to the second processing module in a first-in, first-out manner.

[0035] As another example, in the system for real-time data processing according to the present invention, the processing module is further configured to parse the transaction information data to obtain instantaneous status data, wherein the instantaneous status data includes the latest status data of each order and the latest completion status of each product in the order; and

[0036] The processing module is further configured to aggregate the instantaneous state data to obtain business indicator data;

[0037] The system further includes a first storage module and a second storage module, wherein the first storage module is used to store the business indicator data; and the second storage module is used to store the transient state data.

[0038] As another example, in the system for real-time data processing described in the present invention, the processing module is a data processing module based on Lambda service.

[0039] As another example, in the system for real-time data processing described in the present invention, the judgment module is further used to judge whether the data contains preset flag information, and if so, judge that the data is transaction information data.

[0040] According to another aspect of the present invention, the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the method as described above when the processor is running.

[0041] According to yet another aspect of the present invention, the present invention further provides a storage medium on which program instructions are stored. The program instructions are used to execute the above-mentioned method when running.

[0042] In the present application, after receiving data from the device side, by judging whether the data is transaction information data, if the data is transaction information data, the processing module is called to process the transaction information data in real time. In this way, since the judgment step is added, and the processing module is called only when the judgment result is that the received data is transaction information data, the processing module is not always in working state, thereby saving costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features, and advantages of the present invention will become more apparent through a more detailed description of the embodiments of the present invention with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and are not intended to limit the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0044] Figure 1 A schematic diagram of an application environment of a method for real-time data processing according to an embodiment of the present application is shown schematically;

[0045] Figure 2 A schematic block diagram of a system for real-time data processing according to an embodiment of the present application is schematically shown;

[0046] Figure 3 The following schematically shows a flow chart of a method for real-time data processing according to an embodiment of the present application;

[0047] Figure 4 A schematic block diagram of a system for real-time data processing according to another embodiment of the present application is schematically shown;

[0048] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more apparent, exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0050] The method for real-time data processing provided by this application can be applied to Figure 1In the application environment shown in FIG. 1 , it can be understood that the application referred to here is actually the connection between data sources. For example, the data to be processed in the method for real-time data processing provided in this application is from Figure 1 The device side shown, and Figure 1 The data warehouse shown in the example is used for batch processing. For example, at a fixed time each day, all data before the current day of the week or month is batch processed to obtain batched indicator data, which is then stored in a business database (e.g., a MongoDB database). In the method for real-time data processing provided in this application, real-time processing refers to the real-time processing of specific data received from the device to obtain business indicator data.

[0051] Specifically, if Figure 1 As shown, the device side realizes information interaction with the cloud microservices through the Internet of Things technology. The cloud microservices are the logic developed by the back-end developers using code. For example, the device sends order information to the cloud, and the cloud sends the start of production information to the device. The cloud service sends all business data to the database (such as the business database) for storage, and sends part of the data to the storage device (the storage device and Figure 4 The storage unit S3 in the data warehouse can be the same component or module, for example, both can be AWS's storage unit S3), for analysis by big data engineers building a data warehouse and provide data to be processed for the real-time processing method provided in this application. For batch processing, big data engineers capture data from the storage device for layer-by-layer analysis, and store the final business indicator data in the business database. In one example, the Internet of Things technology and cloud services can be built based on the cloud computing service AWS, and the cloud microservices are deployed on AWS's cloud servers. The business database can be a MongoDB database, or of course other databases. AWS's cloud server sends all business data to the MongoDB database for storage, and at the same time sends part of the data (for example, transaction information data such as order amount, order details, beverage type, discount information, etc.) to AWS's storage unit S3 for analysis by big data engineers building a data warehouse. Big data engineers capture data from the storage unit S3 for analysis, and store the final business indicator data in the MongoDB database. For real-time processing, the specific real-time processing method will be described in detail below.

[0052] Figure 2 and Figure 4 The system block diagram for real-time data processing according to different embodiments of the present application is schematically shown. The method for real-time data processing in the embodiment of the present application can be implemented based on this system architecture.

[0053] Specifically, if Figure 2As shown, in this embodiment, the system includes a receiving module, a judging module, and a processing module. The receiving module is configured to receive data from a device end, wherein the device end is configured to complete the production of goods in an order; the judging module is configured to determine whether the data is transaction information data. If the data is transaction information data, the receiving module sends a call message to the processing module; and the processing module is configured to receive the call message from the receiving module and process the transaction information data based on business indicators to obtain business indicator data, which is used to represent the business indicators.

[0054] Figure 3 FIG. 1 is a schematic flow chart of a method 100 for real-time data processing according to an embodiment of the present invention. Figure 3 As shown, the method 100 for real-time data processing may include the following steps.

[0055] In step S110 , the receiving module receives data from a device end, wherein the device end is used to complete the production of the goods in the order.

[0056] It can be understood that users place orders through mini-programs or ordering software. After the order is placed, the order information is uploaded to the cloud. The cloud sends the order task to the device. After receiving the order task, the device starts to produce the goods in the order (such as drinks). The device sends data related to the order status to the receiving module, and the device also uploads some other data. Specifically, the data received by the receiving module from the device can be considered to be two categories, one is business data, and the other is log record data. Among them, business data mainly includes transaction information data and device information data. Specifically, transaction information data includes order amount, order details, drink type, discount information, etc., and device information data mainly includes device status, material status, etc. Log record data can be understood as some real-time messages, such as when the device is online, a record is generated, and when the device is offline, another record is generated.

[0057] Step S120: the judging module judges whether the data is transaction information data. If the data is transaction information data, the receiving module sends a call message to the processing module.

[0058] As can be seen from the above, the data received from the device side in step 110 includes not only transaction information data, but also device information data, log record data, etc., and in this application, only transaction information data is processed in real time, so it is necessary to determine which of the data received from the device side is transaction information data. In one embodiment, the step of determining whether the data is transaction information data includes: determining whether the data contains preset flag information, and if so, determining that the data is transaction information data. Specifically, if the data received by the receiving module is transaction information data, the data will carry preset flag information, such as an order ID. In this way, when the received data has an order ID, the judgment module can determine that the data is transaction information data.

[0059] For example, the determination module of the system for real-time data processing in this embodiment is further configured to determine whether the data contains a preset flag information, and if so, to determine that the data is transaction information data. The explanation of this embodiment has been described above and will not be repeated here.

[0060] In one embodiment, the receiving module may include a first receiving unit and a second receiving unit, the first receiving unit is used to store data unrelated to transaction information data (such as device information data), the second receiving unit is used to store the transaction information data, and the second receiving unit is bound to a calling relationship with the processing unit. When the second receiving unit receives transaction information data, the receiving module sends a calling information to the processing module.

[0061] In step S130 , the processing module receives the call information and processes the transaction information data based on the business indicator to obtain business indicator data, where the business indicator data is used to represent the business indicator.

[0062] In this application, when the data received from the device side is transaction information data, the transaction information data is processed based on business indicators to obtain business indicator data. For example, business indicators can be of different dimensions, such as the device dimension. For example, there are many devices, and the devices will be leased to different tenants. At this time, it is necessary to obtain business indicators of the tenant dimension. Specifically, for the device dimension, the business indicators here can be understood as the order flow and number of orders completed by device A on the same day, or the number of orders completed by device A on the same day, the total number of cups of beverages, and the total amount of orders; for the tenant dimension, the business indicators here can be understood as the real-time order flow and number of orders of different tenants on the same day, the real-time order flow and number of orders of different tenants in different time periods on the same day (hourly dimension), the number of cups and amount of sales of different beverages of different tenants on the same day in reverse order, the number of sales orders, the number of cups sold, and the amount of sales under the weekly dimension of different tenants, and the number of sales orders, the number of cups sold, and the amount of sales under the monthly dimension of different tenants. The business indicator data is used to characterize the business indicators. It can be understood that the business indicator data is the result of the processing module processing the transaction information data based on the business indicators. For example, for the device dimension, the number of orders completed by device A on that day is 100 orders. The 100 orders here can be considered as business indicator data used to characterize the corresponding business indicators.

[0063] In the present application, after receiving data from the device side, by judging whether the data is transaction information data, if the data is transaction information data, the processing module is called to process the transaction information data in real time. In this way, since the judgment step is added, and the processing module is called only when the judgment result is that the received data is transaction information data, the processing module is not always in working state, thereby saving costs.

[0064] In one embodiment of the present application, step S130 includes the following steps: the processing module receives the call information; after the processing module receives the call information, the transaction information data is marked according to a predetermined dimension to obtain the transaction information data under the predetermined dimension; the processing module processes the transaction information data under the predetermined dimension based on business indicators to obtain business indicator data.

[0065] In the system for real-time data processing provided in an embodiment of the present application, the processing module includes: a first processing module for receiving the call information and, after receiving the call information, marking the transaction information data according to a predetermined dimension to obtain the transaction information data under the predetermined dimension; a second processing module for processing the transaction information data under the predetermined dimension based on business indicators to obtain business indicator data. Exemplarily, the processing module is a data processing module based on Lambda service. Specifically, as Figure 4In the system for real-time data processing provided by the embodiment of the present application shown, the first processing module can be a first-level Lambda service, and the second processing module can be a second-level Lambda service. The receiving module sends a call message to the first-level Lambda service. After receiving the call message, the first-level Lambda service marks the transaction information data according to a predetermined dimension to obtain the transaction information data under the predetermined dimension. The second-level Lambda service processes the transaction information data under the predetermined dimension based on business indicators to obtain business label data. For example, in this embodiment, Figure 4 The storage unit S3 in the embodiment may include the aforementioned receiving module and judgment module. The method and system for real-time data processing provided by the present application are particularly suitable for enterprises with smaller business volumes. Due to the smaller business volumes, in most cases, operators only require real-time data on the day's transaction volume and sales. Therefore, a serverless service such as AWS Lambda is used. The Lambda service is only activated upon receipt of transaction information data, thereby incurring fees. No costs are incurred when there is no data, which helps save costs.

[0066] To facilitate the identification of data attributes, the transaction information data is marked before being processed. For example, for the tenant dimension, the marked data specifically corresponds to which tenant. For the device dimension, the transaction information data can be marked to which device. In this way, if you only focus on the number of orders completed by device A, other devices can be quickly filtered out through marking, thereby improving data processing efficiency.

[0067] In order to ensure the order and integrity of data transmission and to ensure that data is not lost, in a real-time manner, the processing module in the above steps processes the transaction information data under the predetermined dimension based on the business indicators to obtain business marker data, including: the processing module processes the transaction information data under the predetermined dimension according to the first-in-first-out mechanism based on the business indicators to obtain business indicator data. Specifically, the processing module processes the transaction information data under the predetermined dimension according to the first-in-first-out mechanism based on the business indicators of the processing module to obtain business indicator data, including: the processing module inputs the transaction information data under the predetermined dimension obtained after marking into a first-in-first-out queue; based on the business indicators, the transaction information data output from the first-in-first-out queue is processed to obtain business indicator data. In one embodiment, the first-level Lambda service marks the transaction information data according to the predetermined dimension, and inputs the transaction information data under the predetermined dimension obtained after marking into the first-in-first-out queue, and the second-level Lambda service processes the transaction information data output from the first-in-first-out queue to obtain business indicator data.

[0068] In the real-time data processing system provided in an embodiment of the present application, the system includes a message transmission module and a queue console module. The message transmission module is used to broadcast messages corresponding to the transaction information data under the predetermined dimension, and the queue console module is used to receive the messages broadcast by the message transmission module and send the messages to the second processing module according to the first-in-first-out mechanism. The message transmission module can be, for example, the Amazon SNS FIFO service, and correspondingly, the queue console module can be, for example, the Amazon SQS FIFO service. Continue to refer to Figure 4 In this embodiment, the message transmission module and the queue console module are sequentially arranged between the first-level Lambda service and the second-level Lambda service. After receiving the data to be processed, the first-level Lambda service transmits it to the second-level Lambda service via the message transmission module and the queue console module. The processed data by the first-level Lambda service is then delivered to the second-level Lambda service via Amazon FIFO SNS + Amazon FIFO SQS. Specifically, messages are broadcast via Amazon FIFO SNS, allowing downstream users to subscribe to different topics. Messages broadcast by Amazon FIFO SNS are received by Amazon FIFO SQS, forming a message queue. The first-in-first-out mechanism of FIFO SQS ensures message ordering, and configuring Amazon FIFO SQS also ensures that messages are not lost. Furthermore, the number of Amazon FIFO SQS instances depends on the number of subscribed topics. For example, in this embodiment, two Amazon FIFO SQS instances can be subscribed to Amazon FIFO SNS, with one instance servicing the order topic and the other servicing the cup topic.

[0069] Due to the multi-tenant characteristics of the SaaS platform (for example, an enterprise has many devices and leases each device to multiple tenants, and one of the functions of the SaaS platform is to manage each tenant), this application is particularly suitable for situations where the number of tenants increases rapidly in the later stage. As the number of tenants increases, the amount of data will also increase accordingly. Therefore, for this application, if the number of tenants increases, it is only necessary to add the corresponding Amazon FIFO SQS and the correspondence between Amazon FIFO SQS and the processing module. It is precisely because of this strong operability that the method and system provided by this application can be expanded to correspond to multi-tenant scenarios and have good scalability.

[0070] In one embodiment, step 130 processes the transaction information data based on business indicators to obtain business indicator data, including the following steps: parsing the transaction information data to obtain instantaneous status data; parsing the transaction information data to obtain instantaneous status data, wherein the instantaneous status data includes the latest status data of each order and the latest completion status of each product in the order; aggregating the instantaneous status data to obtain business indicator data; the method also includes: storing the business indicator data and the instantaneous status data in different storage modules.

[0071] In the system for real-time data processing provided by this embodiment, the processing module is further used to parse the transaction information data to obtain instantaneous status data, and the instantaneous status data includes the latest status data of each order and the latest completion status of each product in the order; and the processing module is further used to aggregate the instantaneous status data to obtain business indicator data; the system also includes a first storage module and a second storage module, the first storage module is used to store the business indicator data; the second storage module is used to store the instantaneous status data.

[0072] Aggregation processing includes summing and counting, for example, summing the order amounts through the SUM function and counting the order IDs through the COUNT function.

[0073] Specifically, the business indicator data can be stored in the first storage module, and the transient state data can be stored in the second storage module. Preferably, the second storage module is a database based on memory processing that can read and write very quickly. Exemplarily, the business indicator data is stored in a MongoDB database, and the transient state data is stored in a DynamoDB database. Of course, the second storage module can also be other databases, such as a Redis database. On the one hand, based on considerations of cost and actual application scenarios, on the other hand, since the data format of DynamoDB is similar to that of MongoDB, it is not necessary to use too much code to convert the data format during the development process. Therefore, in this embodiment, the first storage module is preferably a MongoDB database, and the second storage module is preferably a DynamoDB database.

[0074] The instantaneous state data includes the latest state data for each order and the latest completion status of each item in the order. Specifically, DynamoDB, as a non-relational database, is primarily used in this embodiment to store intermediate processes. Each order may transition to any intermediate state or final state. For example, at a certain moment, the status of Order A is "In Progress" and the status of Order B is "Paid." At this moment, in the DynamoDB database, the latest status of Order A is "In Progress" and the latest status of Order B is "Paid." It can be understood that the data stored in the DynamoDB database is the latest state data corresponding to the latest status of "In Progress" or "Paid." The following description is similar, and for convenience, it is simply described as a state update. At the next moment, the status of Order A transitions to the "Waiting for Collection" state, and the status of Order A stored in DynamoDB is refreshed to the "Waiting for Collection" state. Similarly, at this moment, the status of Order B transitions to "Completed," and the corresponding status of Order B stored in DynamoDB is "Completed." In other words, the order status and the latest status of each drink in each order stored in DynamoDB may be an intermediate state or a final state. For example, for an order, the order status will generally include the following statuses in sequence: order processing, paid, received, in production, waiting to be collected, completed, discarded, and order refunded.

[0075] Furthermore, this embodiment sets up two databases, rather than storing all processed data in a single database, such as MongoDB or DynamoDB, because MongoDB, as the business database for all retail projects, stores all business information. Developers who need to extract data simply connect to the database's interface (for example, to obtain the corresponding data through the interface for front-end colleagues to display). Temporary order status data, however, is not used as final display data and has no use value to other developers. Therefore, a distinction is made in the functional definition, and DynamoDB is chosen. This provides clear functional division, strong operability, and convenient data access.

[0076] Combine Figure 4 In the system shown, the role of the second storage module is to facilitate aggregate statistics. The sales amount at a certain moment must be the sales amount of all orders accumulated that day. If there is no data in a previous time period, both the first-level and second-level Lambda will be closed. At this time, the second storage module is needed to store this part of the data, otherwise it will be lost, resulting in inaccurate results.

[0077] In one embodiment, the first-level Lambda service tags the transaction information data according to predetermined dimensions, and sends the marked transaction information data under the predetermined dimensions to the second-level Lambda through Amazon FIFO SNS and Amazon FIFOSQS in sequence. The second-level Lambda parses the marked transaction information data under the predetermined dimensions to obtain instantaneous state data, stores the instantaneous state data in the second storage module, and performs aggregation processing based on the instantaneous state data stored in the second storage module to obtain business indicator data, and stores the business indicator data in the first storage module.

[0078] According to yet another aspect of the present invention, an electronic device is provided. Figure 5 FIG. 1 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Figure 5 As shown, the electronic device includes a processor 310 and a memory 320. The memory 320 stores computer program instructions, which are used by the processor 310 to execute the method 100 for real-time data processing as described above when the computer program instructions are executed.

[0079] According to another aspect of the present invention, a storage medium is also provided. Program instructions are stored on the storage medium, and the program instructions are used to execute the method 100 for real-time data processing as described above when running. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0080] A person skilled in the art can understand the specific implementation scheme of the above panel detection device, electronic device and storage medium by reading the above description of the panel detection method. For the sake of brevity, it is not repeated here.

[0081] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present invention. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as claimed in the appended claims.

[0082] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical function division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not performing some features.

[0084] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0085] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to the present invention should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following a specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0086] Those skilled in the art will understand that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings), as well as all processes or units of any method or apparatus disclosed herein, may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that provides the same, equivalent, or similar purpose.

[0087] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0088] The various component embodiments of the present invention may be implemented in hardware, as software modules running on one or more processors, or as a combination thereof. Those skilled in the art will appreciate that, in practice, a microprocessor or digital signal processor (DSP) may be used to implement some or all of the functions of some modules in the panel detection device according to the embodiments of the present invention. The present invention may also be implemented as a device program (e.g., a computer program or computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium or in the form of one or more signals. Such signals may be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0089] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0090] The foregoing descriptions are merely specific embodiments of the present invention, or illustrations of specific embodiments. The scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein are intended to be encompassed within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for real-time data processing in commodity production equipment, applied to a real-time data processing system, characterized in that: The system includes a receiving module, a judging module, and a processing module, and the method includes: The receiving module receives data from a device, wherein the device is used to complete the production of the goods in the order; The judging module judges whether the data is transaction information data, and if the data is transaction information data, the receiving module sends the call information to the processing module; The processing module receives the call information. After receiving the call information, the processing module tags the transaction information data according to a predetermined dimension to obtain transaction information data under the predetermined dimension; the processing module processes the transaction information data under the predetermined dimension based on the business indicator to obtain business indicator data, wherein the business indicator data is used to represent the business indicator; The processing of the transaction information data based on the business indicator to obtain the business indicator data includes: Parsing the transaction information data to obtain instantaneous status data, wherein the instantaneous status data includes the latest status data of each order and the latest completion status of each product in the order; Aggregating the instantaneous state data to obtain business indicator data; The method further comprises: The business indicator data and the instantaneous state data are stored in different storage modules.

2. The method for real-time data processing for commodity production equipment according to claim 1, characterized in that: The processing module processes the transaction information data under the predetermined dimension based on the business indicator to obtain business indicator data, including: The processing module processes the transaction information data under the predetermined dimension based on the business indicators according to a first-in-first-out mechanism to obtain business indicator data.

3. The method for real-time data processing for commodity production equipment according to claim 2, characterized in that: The processing module processes the transaction information data under the predetermined dimension according to a first-in-first-out mechanism based on the business indicator to obtain business indicator data, including: The processing module inputs the marked transaction information data under the predetermined dimension into a first-in-first-out queue; Based on the business indicators, the transaction information data output from the first-in-first-out queue is processed to obtain business indicator data.

4. The method for real-time data processing for commodity production equipment according to claim 1, characterized in that: The storing of the business indicator data and the instantaneous state data in different storage modules includes: The business indicator data is stored in a MongoDB database, and the transient state data is stored in a DynamoDB database.

5. The method for real-time data processing for commodity production equipment according to claim 1, characterized in that: The determining whether the data is transaction information data includes: determining whether the data contains preset flag information; if so, determining that the data is transaction information data.

6. A system for real-time data processing of commodity production equipment, characterized in that: The system comprises: A receiving module, configured to receive data from a device, wherein the device is configured to complete the production of the goods in the order; a judging module, configured to judge whether the data is transaction information data; if the data is transaction information data, the receiving module sends a call message to the processing module; a processing module, configured to receive the call information from the receiving module, and after receiving the call information, mark the transaction information data according to a predetermined dimension to obtain transaction information data under the predetermined dimension; the processing module is further configured to process the transaction information data under the predetermined dimension based on a business indicator to obtain business indicator data, wherein the business indicator data is used to represent the business indicator; The processing module is further configured to parse the transaction information data to obtain instantaneous status data, wherein the instantaneous status data includes the latest status data of each order and the latest completion status of each product in the order; and The processing module is further configured to aggregate the instantaneous state data to obtain business indicator data; The system further includes a first storage module and a second storage module, wherein the first storage module is used to store the business indicator data; and the second storage module is used to store the transient state data.

7. The system for real-time data processing of commodity production equipment according to claim 6, characterized in that: The processing module includes: a first processing module, configured to receive the call information, and after receiving the call information, mark the transaction information data according to a predetermined dimension to obtain the transaction information data under the predetermined dimension; The second processing module is used to process the transaction information data under the predetermined dimension based on the business indicator to obtain business indicator data.

8. The system for real-time data processing of commodity production equipment according to claim 7, characterized in that: The system includes a message transmission module and a queue console module. The message transmission module is used to broadcast messages corresponding to the transaction information data under the predetermined dimension. The queue console module is used to receive the messages broadcast by the message transmission module and send the messages to the second processing module according to the first-in-first-out mechanism.

9. The system for real-time data processing of commodity production equipment according to claim 6, characterized in that: The processing module is a data processing module based on Lambda service.

10. The system for real-time data processing of commodity production equipment according to claim 6, characterized in that: The judgment module is further configured to judge whether the data contains preset flag information, and if so, judge that the data is transaction information data.

11. An electronic device comprising a processor and a memory, wherein: The memory stores computer program instructions, which are used by the processor to execute the method according to any one of claims 1 to 5 when executed. 12 . A storage medium storing program instructions, wherein the program instructions are used to execute the method according to claim 1 when running.

Images