Logistics In-and-Out Station Efficiency Monitoring Method, Device, Equipment and Storage Medium

Abstract

The present invention relates to the field of logistics big data, and discloses a method, device, equipment and storage medium for monitoring the efficiency of logistics in and out of stations. The method includes: receiving the data stream real-time input for logistics in and out of stations, and performing aggregation and duplicate removal processing to obtain compressed data; using a preset asynchronous method to distributively import the compressed data into a preset real-time data table to obtain logistics detail data, and updating a preset offline data table to obtain updated logistics index information; calculating the in and out efficiency slices of logistics in and out of stations by using the logistics detail data in the real-time data table, and statistically calculating the in and out status information of logistics in and out of stations by using the logistics index information in the offline data table; calculating multiple efficiency indexes of logistics in and out of stations according to the in and out efficiency slices and the in and out status information, summarizing each efficiency index into a materialized view and displaying it at the logistics in and out of stations. The present invention improves the consistency of real-time data and offline data storage for multiple departments, as well as the timeliness of data storage.

Description

Technical Field

[0001] The present invention relates to the field of logistics big data, and in particular to a method, device, equipment and storage medium for monitoring the efficiency of logistics in and out of stations. Background Art

[0002] With the rapid development of the logistics industry and the continuous expansion of logistics companies, a large amount of logistics data is generated in daily work, including tens of billions of scanned data, multiple detailed data sources, and a short recalculation cycle. The front-end interface needs to respond in real time to meet business requirements. Therefore, the processing of back-end data needs to meet the timeliness of real-time or near-real-time data batch processing, and the data accuracy needs to be very high. The error between real-time data and offline data needs to be very small to maintain the operation of daily logistics business.

[0003] The existing logistics data involves multiple business terminals, and it is difficult to keep the calibers of different corresponding indicators consistent to ensure data consistency. Facing the real-time writing of massive data, it is necessary to ensure the stable execution of complex logic and provide efficient interfaces externally at the same time. Different operation logics need to be adopted for different data tables to remove duplicates, and the timeliness of data needs to be ensured. There are problems with poor timeliness in the existing methods for processing massive logistics data, and there are also problems with insufficient timeliness in the data processing of logistics in and out of stations. Summary of the Invention

[0004] The main purpose of the present invention is to solve the technical problem of insufficient timeliness in the existing methods for processing massive logistics data of logistics in and out of stations.

[0005] In a first aspect of the present invention, a method for monitoring the efficiency of logistics in and out of stations is provided, including: receiving a data stream real-time input for logistics in and out of stations, and performing aggregation and duplicate removal processing on the data stream to obtain compressed data; using a preset asynchronous method to distributively import the compressed data into a preset real-time data table, and using the real-time data table to update a preset offline data table, where the data table stores logistics detailed data, and the offline data table stores logistics index information; calculating the in-out efficiency slices of logistics in and out of stations using the logistics detailed data in the real-time data table, and statistically calculating the in-out status information of logistics in and out of stations using the logistics index information in the offline data table; calculating multiple efficiency indicators of logistics in and out of stations according to the in-out efficiency slices and the in-out status information, and summarizing each efficiency indicator into a materialized view and displaying it at the logistics in and out of stations.

[0006] Optionally, in a first implementation manner of the first aspect of the present invention, before using the real-time data table to update the preset offline data table, it further includes: obtaining the offline data of logistics in and out of stations, and loading the offline data into a preset configuration table to obtain logistics configuration information; writing the logistics configuration information into the preset offline data table to update the logistics index information in the offline data table.

[0007] Optionally, in the second implementation manner of the first aspect of the present invention, the aggregating and deduplicating the data stream to obtain compressed data includes: extracting a plurality of attribute identification information and attribute parameters corresponding to the attribute identification information in the data stream, and performing an aggregation process on each of the attribute identification information to obtain aggregated identification information; querying a preset identification reference table by using the aggregated identification information, and judging whether the aggregated identification information exists in the identification reference table according to the query result; if it exists, removing each attribute parameter corresponding to the aggregated identification information; if it does not exist, performing an aggregation process on each attribute parameter corresponding to the aggregated identification information to obtain compressed data.

[0008] Optionally, in the third implementation manner of the first aspect of the present invention, the step of distributively importing the compressed data into a preset real-time data table by using a preset asynchronous method further includes: if the preset asynchronous method is an asynchronous thread method, calling a corresponding number of threads from a preset thread pool for each storage queue according to the number of storage queues of the compressed data in a preset distributed message queue; asynchronously importing the compressed data sequentially stored in each storage queue into the preset real-time data table according to the structure data storage protocol of the thread.

[0009] Optionally, in the fourth implementation manner of the first aspect of the present invention, the step of distributively importing the compressed data into a preset real-time data table by using a preset asynchronous method further includes: if the preset asynchronous method is an asynchronous request method, respectively generating data storage requests by using the compressed data sequentially stored in each storage queue of the preset distributed message queue through a preset transmission protocol; asynchronously sending the data storage requests to a storage system where the preset real-time data table is located through each storage queue; receiving response information of the storage system to each data storage request, and determining an import result of the compressed data according to the response information, where the import result includes import success and import failure.

[0010] Optionally, in the fifth implementation manner of the first aspect of the present invention, the step of summarizing each of the performance indicators into a materialized view and displaying the materialized view includes: storing each of the performance indicators in a corresponding storage location in a preset materialized view framework according to the indicator type of each of the performance indicators to obtain a materialized view;

[0011] querying, in the materialized view, a performance indicator mapped to the interface information according to the interface information of a front-end interface in the logistics in-and-out station, and obtaining a corresponding performance indicator from the logistics view and displaying the performance indicator on the front-end interface according to the query result.

[0012] The second aspect of the present invention provides a performance monitoring device for logistics in and out of the station, including: a deduplication module, which is used to receive a data stream entered in real time for logistics in and out of the station, and aggregate and deduplicate the data stream to obtain compressed data; an import module, which is used to use a preset asynchronous method to distribute the compressed data into a preset real-time table, and use the real-time table to update a preset offline table, wherein the table stores logistics detail data and the offline table stores logistics indicator information; a calculation module, which is used to use the logistics detail data in the real-time table to calculate the in-and-out efficiency slices of logistics in and out of the station, and use the logistics indicator information in the offline table to count the in-and-out status information of logistics in and out of the station; a summary module, which is used to calculate multiple performance indicators of logistics in and out of the station based on the in-and-out efficiency slices and the in-and-out status information, summarize each of the performance indicators into a materialized view and display it at the logistics in and out station.

[0013] Optionally, in a first implementation method of the second aspect of the present invention, the logistics entry and exit efficiency monitoring device also includes a loading module, which is used to: obtain offline data of logistics entry and exit, and load the offline data into a preset configuration table to obtain logistics configuration information; write the logistics configuration information into a preset offline table to update the logistics indicator information in the offline table.

[0014] Optionally, in a second implementation method of the second aspect of the present invention, the deduplication module includes: an identification aggregation unit, used to extract multiple attribute identification information and attribute parameters corresponding to the attribute identification information in the data stream, and aggregate each of the attribute identification information to obtain aggregated identification information; a query unit, used to use the aggregated identification information to query a preset identification reference table, and determine whether the aggregated identification information exists in the identification reference table based on the query result; a data aggregation unit, used to remove the corresponding attribute parameters in the aggregated identification information if they exist; if not, aggregate the corresponding attribute parameters in the aggregated identification information to obtain compressed data.

[0015] Optionally, in a third implementation of the second aspect of the present invention, the import module includes: an asynchronous thread import unit, which is used to call a corresponding number of threads in each storage queue from the preset thread pool according to the number of storage queues of the compressed data in the preset distributed message queue if the preset asynchronous method is an asynchronous thread mode; and asynchronously import the compressed data sequentially stored in each storage queue into a preset real-time table according to the structural data storage protocol of the thread.

[0016] Optionally, in the fourth implementation manner of the second aspect of the present invention, the import module further includes: an asynchronous request import unit, configured to, if the preset asynchronous method is an asynchronous request mode, sequentially store the compressed data in each storage queue of the preset distributed message queue, generate data storage requests respectively through the preset transmission protocol; asynchronously send the data storage requests to the storage system where the preset real-time data table is located through each of the storage queues; receive the response information of the storage system to each of the data storage requests, and determine the import result of the compressed data according to the response information, where the import result includes import success and import failure.

[0017] Optionally, in the fifth implementation manner of the second aspect of the present invention, the summary module includes: a storage unit, configured to store each of the performance indicators at a corresponding storage location in the preset materialized view framework according to the indicator type of each of the performance indicators, to obtain a materialized view; a display unit, configured to query the performance indicators mapped to the interface information in the materialized view according to the interface information of the front-end interface in the logistics in and out station, and obtain the corresponding performance indicators from the logistics view according to the query result for display on the front-end interface.

[0018] The third aspect of the present invention provides a performance monitoring device for logistics in and out station, including: a memory and at least one processor, where instructions are stored in the memory; the at least one processor invokes the instructions in the memory to enable the performance monitoring device for logistics in and out station to execute the steps of the above-mentioned performance monitoring method for logistics in and out station.

[0019] The fourth aspect of the present invention provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are run on a computer, the computer is enabled to execute the steps of the above-mentioned performance monitoring method for logistics in and out station.

[0020] In the technical solution provided by the present invention, among the data sources generated during the logistics in and out of the station, it includes the real-time input data stream and the offline data. Among them, before the real-time input data stream is imported into the real-time data table, the data stream is aggregated and de-duplicated through a distributed queue to ensure that the real-time data table can be obtained immediately after import, reducing the database pressure and the subsequent calculation amount of the database; the offline data is directly loaded into the offline data table to directly form the corresponding physical indicators. Through the logistics detail data in the real-time data table, the in and out efficiency slices of the physical in and out of the station can be directly calculated. By combining the physical detail data and the logistics index information in the offline data table, new physical index information can be obtained, and then the in and out status information of the logistics in and out of the station can be calculated to obtain the basic data for measuring the efficiency of the logistics in and out of the station. Finally, the in and out efficiency slices and the in and out status information are calculated for the materialized view, improving the storage timeliness and query efficiency of the in and out of the station logistics data, and at the same time improving the real-time display of the efficiency index of the logistics in and out of the station, ultimately enabling the overall batch running timeliness of the data generated by the logistics in and out of the station. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the first embodiment of the efficiency monitoring method for the logistics in and out of the station in the embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of the second embodiment of the efficiency monitoring method for the logistics in and out of the station in the embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of the third embodiment of the efficiency monitoring method for the logistics in and out of the station in the embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of an embodiment of the efficiency monitoring device for the logistics in and out of the station in the embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of another embodiment of the efficiency monitoring device for the logistics in and out of the station in the embodiment of the present invention;

[0026] Figure 6 It is a schematic diagram of an embodiment of the efficiency monitoring device for the logistics in and out of the station in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] An embodiment of the present invention provides a method, device, equipment and storage medium for monitoring the efficiency of logistics in and out of a station. The method receives the data stream real-time input for logistics in and out of the station, and performs aggregation and duplicate removal processing to obtain compressed data; uses a preset asynchronous method to distributively import the compressed data into a preset real-time data table to obtain logistics detailed data, and updates a preset offline data table to obtain updated logistics index information; calculates the in-out efficiency slices of logistics in and out of the station using the logistics detailed data in the real-time data table, and statistically analyzes the in-out status information of logistics in and out of the station using the logistics index information in the offline data table; calculates multiple efficiency indicators of logistics in and out of the station according to the in-out efficiency slices and the in-out status information, summarizes each efficiency indicator into a materialized view and displays it at the logistics in and out of the station. The present invention improves the consistency of real-time data and offline data storage for multiple departments, as well as the timeliness of the stored data.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order different from that illustrated or described herein. In addition, the term "comprising" or "having" and any variation thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or equipment comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these process, method, product or equipment.

[0029] For ease of understanding, the specific process of the embodiment of the present invention will be described below. Please refer to Figure 1 The first embodiment of the method for monitoring the efficiency of logistics in and out of the station in the embodiment of the present invention includes:

[0030] 101. Receive the data stream real-time input for logistics in and out of the station, and perform aggregation and duplicate removal processing on the data stream to obtain compressed data;

[0031] It can be understood that the execution subject of the present invention can be a device for monitoring the efficiency of logistics in and out of the station, or a terminal or a server. Specifically, it is not limited here. The embodiment of the present invention takes the server as the execution subject as an example for illustration.

[0032] In this embodiment, for the data stream of real-time entry of logistics in and out of the station, there may be duplicate entries. The conventional processing method is to store it in the database and then perform deduplication. Here, the data stream is directly deduplicated before being stored in the database. Moreover, to ensure the deduplication efficiency of the data stream, the method of aggregation deduplication is used to deduplicate the data stream. Among them, the data stream includes one-dimensional code information for the express delivery scanned into the warehouse by the salesman, vehicle in and out of the station information, express delivery order entry information, logistics prediction information, etc.

[0033] In addition, for the aggregation deduplication processing of the data stream, each data in the data stream is first divided into two parts. One part is the identification information of each data, and the other part is the specific parameters of the data. In the first aggregation, the identification information of each data is first aggregated to obtain a key value for deduplication. By querying whether there is data with the same key value, the deduplication of the data stream can be achieved. In the second aggregation, each data corresponding to the identification information in the key value is aggregated, and the compressed data can be obtained and stored and processed subsequently.

[0034] 102. Adopt a preset asynchronous method to distribute and import the compressed data into a preset real-time data table, and use the real-time data table to update the preset offline data table, where the data table stores logistics detail data, and the offline data table stores logistics index information;

[0035] In this embodiment, by using an asynchronous method to import the compressed data, the import efficiency of the compressed data can be improved and the computing pressure on the database can be reduced. Through a distributed system to organize the compressed data, different compressed data are placed into the corresponding storage queues, and various types of compressed data are asynchronously and orderly imported into the preset real-time data table. After import, the compressed data is sorted into logistics detail data. Moreover, when the compressed data is imported into the real-time data table, through the conventional screening logic, the target logistics detail data in the real-time data table is screened out and written into the offline data table in real time or the update time is set, such as updating every 5 minutes, and new logistics index information can be obtained.

[0036] Specifically, the distributed system can adopt the Kafka message queue. Through the subscription method, various types of compressed data are distributed to the corresponding message queues, and each compressed data is processed asynchronously and orderly, and the compressed data is imported into the real-time data table to obtain logistics detail data, improving the import efficiency of the real-time data table. Among them, the logistics detail data can include unloading details, loading details, center in and out details, order details, and detail prediction information, etc.

[0037] In addition, the data real-time data table is at the DWD (data warehouse detail) layer, which is an isolation layer after the data stream is deduplicated and compressed and before the compressed data is stored in the database. The compressed data is first preliminarily sorted and not stored.

[0038] Specifically, for the logistics detail data in the real-time data table, it is further written into the offline data table to update the logistics index information in the offline data table. Compared with the data entered in real time, the offline data table also adds relevant information such as vehicles, order entry, and prediction, and has the key information required for the current business of the central inbound and outbound stations.

[0039] Specifically, for the generation of the original logistics index information in the offline data table, the offline parameters of logistics in and out set in advance can be directly obtained from structured warehouse data tools such as MySQL or Hive, and the offline parameters can be sorted through a configuration table to obtain the logistics index information.

[0040] 103. Use the logistics detail data in the real-time data table to calculate the inbound and outbound efficiency slices of logistics in and out of the station, and use the logistics index information in the offline data table to count the inbound and outbound status information of logistics in and out of the station;

[0041] In this embodiment, the logistics detail data and logistics index information are initially sorted in the real-time data table and the offline data table. Here, the logistics detail data and logistics index information are further integrated to obtain the basic data for measuring the current efficiency of the central inbound and outbound stations, that is, the inbound and outbound efficiency slices and the inbound and outbound status information. The format of the logistics detail data and logistics index information used here is Bitmap, so there is no need to perform an additional group by on the data.

[0042] Among them, using the data slicing technology, the logistics detail data of the target type is evenly distributed to different computing nodes, and the corresponding inbound and outbound efficiency slices can be obtained by calculating the logistics detail data through a pre-set algorithm; the inbound and outbound efficiency slices are related to the inbound and outbound volume and time of logistics in the logistics detail data. The ratio of the inbound and outbound volume to the inbound and outbound time of different logistics in and out types is the air inbound and outbound efficiency slice. The various logistics in and out types are partitioned to achieve horizontal expansion of data calculation, effectively reducing the access load of each computing node and improving the data access speed and concurrency. For example, the unloading efficiency slice can be calculated by the ratio of the number of unloaded vehicles to the time used for unloading.

[0043] Among them, through the central inbound and outbound station model, the logistics index information can be used to count the inbound and outbound status information, including the progress date, central code, order number, update time, inbound scan time, outbound scan time, inbound identifier, last operation identifier, prediction of the next station's inbound time, etc. For example, through the logistics index information corresponding to the loading and unloading details and the vehicle inbound and outbound details, the inbound and outbound scan times are counted, and the logistics index information corresponding to the cleaning time configuration and the departure time configuration is used to count the departure time. The next station's route is counted through the route configuration. There is no specific limitation here, and the central inbound and outbound station model can be configured according to business requirements.

[0044] Preferably, the in-out efficiency slices may include: real-time platform status slices, unloading efficiency slices, loading efficiency slices, transfer-out efficiency slices, etc., and the in-out status information may include ticket loss, ticket timeliness, goods-drop status, etc. The in-out efficiency slices and the in-out status information are stored in the DWS (Data Warehouse Service, service data layer) layer of the database, integrating offline data and real-time input data streams into different subject domains, and generally using wide tables for storage.

[0045] 104. Calculate multiple performance indicators for the logistics in-out stations based on the in-out efficiency slices and the in-out status information, summarize each performance indicator into a materialized view, and display it at the logistics in-out stations.

[0046] In this embodiment, further calculate the in-out efficiency slices and the in-out status information to obtain the final performance indicators for measuring the in-out of the center, which are divided into two calculation methods: in-out efficiency type and in-out summary statistics. Among them, the performance indicators calculated by the in-out efficiency type are used to measure the in-out efficiency of the center in-out stations, and the performance indicators calculated by the in-out summary statistics are used to measure the in-out carrying capacity of the center stations.

[0047] Specifically, according to the calculation variables of the preset algorithm, select the required in-out efficiency slices and calculate the corresponding performance indicators; according to the statistical variables of the preset algorithm, select the required in-out status information and calculate the corresponding performance indicators. For example, select the order efficiency slice to calculate the matrix efficiency of logistics picking, and select the number of lost tickets and the total number of tickets for loss statistics.

[0048] Preferably, the performance indicators may include matrix efficiency, platform efficiency, employee efficiency, loading and unloading progress, etc., as well as loss statistics, delay statistics, goods-drop statistics, etc. This type of data is stored in the ADS (Application Data Store) layer and summarized in the form of a materialized view, enabling data to be updated synchronously and improving query efficiency.

[0049] In addition, a real-time platform status model can also be used to map the performance indicators in the materialized view to the digital twin main dashboard of the logistics in-out stations for the current center in-out, so as to facilitate real-time monitoring and scheduling by management personnel.

[0050] Specifically, the materialized view directly provides an external empty interface to update the performance of the center in-out stations. The large screen displays the changes of the platform in real time, and the display time levels can be set, including real-time platform status - second level, vehicle detailed information - minute level, vehicle loading and unloading progress - minute level, employee operation efficiency - minute level, throughput - minute level, etc.

[0051] In the embodiment of the present invention, the data source generated by the logistics in and out of the station includes real-time input data stream and offline data, wherein before the real-time input data stream is imported into the real-time table, the data stream is aggregated and deduplicated through the distributed queue to ensure that the real-time table is imported as is, reducing the database pressure and the amount of subsequent database calculations; the offline data is directly loaded into the offline table to directly form the corresponding physical indicators. Through the logistics detail data in the real-time table, the in-and-out efficiency slice of the physical in-and-out station can be directly calculated, and the new physical indicator information is obtained by combining the physical detail data and logistics indicator information in the offline table, and the in-and-out status information of the logistics in and out of the station can be calculated to obtain the basic data for measuring the efficiency of the logistics in and out of the station. Finally, the materialized view of the in-and-out efficiency slice and the in-and-out status information is calculated to improve the storage timeliness and query efficiency of the in-and-out logistics data, and at the same time improve the real-time display of the logistics in-and-out efficiency indicators, and finally make the overall batch timeliness of the data generated by the logistics in and out of the station.

[0052] See also Figure 2 The second embodiment of the method for monitoring the efficiency of logistics in and out of a station in an embodiment of the present invention includes:

[0053] 201. Receive the data stream of real-time entry of logistics in and out of the station;

[0054] 202. Extract multiple attribute identification information and attribute parameters corresponding to the attribute identification information in the data stream, and aggregate the attribute identification information to obtain aggregate identification information;

[0055] 203. Use the aggregated identification information to query a preset identification reference table, and determine whether the aggregated identification information exists in the identification reference table according to the query result;

[0056] 204. If it exists, remove the corresponding attribute parameters in the aggregation identification information; otherwise, aggregate the corresponding attribute parameters in the aggregation identification information to obtain compressed data;

[0057] In this embodiment, the attribute identification information and corresponding attribute parameters in the data stream are extracted, such as the attribute identification information and corresponding attribute parameters of the order number, including the sender's name, address, telephone number and the recipient's name, address, telephone number, as well as the express type, freight, etc. The identification information entered in the past is recorded as a record identification reference table through Redis. When the aggregated identification information and the identification reference table exist, the corresponding data can be determined and deduplication can be performed.

[0058] Specifically, for example, the identification information of the data includes: central inbound / outbound ID1, loading / unloading platform ID2, processing clerk ID3, departure voucher ID4, and order number ID5. Then, aggregate each piece of identification information to obtain the key value: ID1_ID2_ID3_ID4_ID5, and determine whether there is already the same key value ID1_ID2_ID3_ID4_ID5 to determine whether there is the same data. Then, for ID1_ID2_ID3_ID4_ID5, aggregate the corresponding central inbound / outbound details, loading / unloading platform information, identity information of the processing clerk, goods loading / unloading information corresponding to the departure voucher, and specific logistics parameters in the order number, and the corresponding compressed data can be obtained.

[0059] Preferably, construct the execution process of aggregating and deduplicating the data stream into a platform efficiency model to automatically deduplicate the data stream before writing. The data stream can adopt the SUM aggregation type in the Aggregate model and use Redis for aggregation and deduplication, and the obtained compressed data format is Bitmap.

[0060] 205. Obtain the offline data of the logistics inbound / outbound, and load the offline data into a preset configuration table to obtain the logistics configuration information.

[0061] 206. Write the logistics configuration information into the preset offline data table to update the logistics index information in the offline data table.

[0062] In this embodiment, for the offline data, it can be stored through the Duplicate data model, without setting a primary key and without aggregation. The offline data can be directly loaded into the configuration table for configuration.

[0063] Specifically, if the data source is in Hive, the offline data can be loaded in the way of Broker Load. If the data source is in MySQL, the offline data can be loaded in the way of ODBC. In addition, the offline data can include fixed-set data such as the inbound / outbound vehicle frequency, routing information, and driving route of the central inbound / outbound.

[0064] In this embodiment, through the preset configuration table, obtain the original offline data for preliminary division to obtain the logistics configuration information with a preset configuration structure, including departure time, cleaning frequency, routing configuration, etc. Then, write the logistics configuration information into the offline data table, and the logistics configuration information can be further divided to obtain the logistics index information, which can include lost parcels, parcel timeliness, goods drop-off status, vehicle inbound / outbound time, outbound prediction time, and next stop inbound time, etc.

[0065] 207. Adopt a preset asynchronous method to distributively import compressed data into a preset real-time data table, and use the real-time data table to update the preset offline data table, where the data table stores logistics detail data and the offline data table stores logistics index information;

[0066] 208. Use the logistics detail data in the real-time data table to calculate the in-out efficiency slices of logistics in and out of the station, and use the logistics index information in the offline data table to count the in-out status information of logistics in and out of the station;

[0067] 209. Calculate multiple performance indicators of logistics in and out of the station based on the in-out efficiency slices and the in-out status information, and store each performance indicator in the corresponding storage location in the preset materialized view framework according to the indicator type of each performance indicator to obtain a materialized view;

[0068] 210. Query the performance indicators mapped to the interface information in the materialized view according to the interface information of the front-end interface in the logistics in and out of the station, and obtain the corresponding performance indicators from the logistics view according to the query result for display in the front-end interface.

[0069] In this embodiment, the materialized view converts the data view or SQL information into physical data storage and provides different refresh strategies for performance indicators: whether to refresh regularly or in a timely manner, incremental refresh or global refresh, etc., which can be selected according to the actual situation.

[0070] Specifically, the materialized view is called the master table (during replication) or the detail table (in the data warehouse). For replication, the materialized view allows maintaining a copy of the performance indicators locally, and the copy is only for reading. If you want to modify the local copy, you must use the advanced replication function. When extracting performance indicators from a table or view, you can extract them from the materialized view.

[0071] For the data warehouse, the created materialized view is usually an aggregate view, a single-table aggregate view, and a join view. In a replication environment, the created materialized view includes the primary key, RowID, and subquery view of the performance indicators. Since the materialized view physically exists, indexes can be created for each performance indicator.

[0072] In the embodiment of the present invention, when calculating the performance indicators of the center in and out of the station based on real-time data and offline data, the materialized view realizes the implementation unity of the summary time and the base table data. When querying, the materialized view table is automatically named, which improves the calculation and query efficiency compared with the storage method of ordinary views; in addition, the Bitmap data is used as the basic data for calculation, analysis, and statistics, which can perform accurate deduplication within a large range, improving the calculation efficiency compared with other data formats, and deduplication has been realized before storage.

[0073] Please refer to Figure 3, the third embodiment of the efficiency monitoring method for logistics in and out of the station in the embodiments of the present invention includes:

[0074] 301. Receive the data stream real-time input for logistics in and out of the station, and perform aggregation and duplicate removal processing on the data stream to obtain compressed data;

[0075] 302. If the preset asynchronous method is the asynchronous thread method, then according to the number of storage queues in the preset distributed message queue for the compressed data, call the corresponding number of threads from the preset thread pool to each storage queue;

[0076] 303. According to the structure data storage protocol of the thread, asynchronously import the compressed data stored in sequence in each storage queue into the preset real-time data table;

[0077] In this embodiment, the data processing architecture for the center in and out of the station can specifically adopt the Doris architecture, including three layers: a database storage tool, FE (front-end node), and BE (back-end node). The database storage tool can include Kafka, Hive, MySQL, etc. The FE is responsible for formulating query plans for data parsing, generation, and scheduling, and the BE is responsible for executing the corresponding query plans and storing data. Both the FE and BE have linear scalability and can be expanded according to requirements.

[0078] The import of the compressed data here is for the BE layer. Among them, the asynchronous method can include the asynchronous thread method and the asynchronous request method. If it is the asynchronous thread method here, the Uniq data model can be adopted. Specifically, through the Routine Load method, directly import the compressed data stored in each storage queue in Kafka into the real-time data table in sequence, and can perform custom monitoring and automatic recovery on the imported data.

[0079] Specifically, the structure data storage protocol is adopted here, such as the MySQL protocol, to store and partition the data required in the real-time data table. Routine Load (routine import) directly imports the compressed data one by one into the real-time data table through the structure data storage protocol. Submit a routine import job through the MySQL protocol to generate a resident thread, continuously read the compressed data from the data source (such as Kafka), and import it into the real-time data table through the SQL syntax in the MySQL protocol.

[0080] 304. If the preset asynchronous method is the asynchronous request method, then respectively use the compressed data stored in sequence in each storage queue in the preset distributed message queue to generate data storage requests through the preset transmission protocol;

[0081] 305. Through each storage queue, asynchronously send data storage requests to the storage system where the preset real-time data table is located;

[0082] 306. Receive response information of the storage system to each data storage request, and determine the import result of the compressed data according to the response information, wherein the import result includes import success and import failure;

[0083] In this embodiment, if an asynchronous request method is used, the compressed data can be imported into the real-time data table through the data stream of the Aggregate data model. Specifically, the compressed data can be imported through the Stream Load method in combination with a customized Flink and a transmission protocol, such as the HTTP protocol.

[0084] The data model application of Aggregate can specifically include the following:

[0085] The aggregation type is sum, which is suitable for ticket quantity statistics. When writing with Flink, it uses Redis to deduplicate.

[0086] The aggregation type is min, which is suitable for recording the earliest entry and exit time;

[0087] The aggregation type is max, which is applicable to the latest entry and exit time, large package mark and maximum weight mark;

[0088] The aggregation type is replace-if_not_null, which is applicable to the last updated record of the data;

[0089] The aggregation type is bitmap, which is used for deduplication statistics of ticket quantity and time slice calculation. The deduplication accuracy of integer data is 100%, which is suitable for deduplication statistics of order numbers.

[0090] In addition, when developing Doris, because the data is aggregated and deduplicated before being stored in the materialized view, the amount of data is less than that of conventional data frameworks, such as Greenplum or Kudu. There is no need for repeated operations and calculations of a large number of row_number_join operations like Greenplum, and there is no need to use Flink combined with Redis for a large number of calculations like Kudu, which reduces the difficulty of Doris development.

[0091] By using the Doris framework to manipulate data streams, a large amount of computing logic in Flink is reduced, reducing investment in development and maintenance, and the computing resources of Flink and Redis can also be released.

[0092] 307. Use a real-time table to update a preset offline table, wherein the table stores logistics detail data and the offline table stores logistics index information;

[0093] 308. Calculate the inbound and outbound efficiency slices of logistics for inbound and outbound at the logistics station using the logistics detail data in the real-time data table, and count the inbound and outbound status information of logistics for inbound and outbound using the logistics index information in the offline data table.

[0094] 309. Calculate multiple efficiency indicators for logistics inbound and outbound based on the inbound and outbound efficiency slices and the inbound and outbound status information, summarize each efficiency indicator into a materialized view and display it at the logistics inbound and outbound.

[0095] In the embodiments of the present invention, through the data model of Doris, when importing compressed data into the real-time data table, asynchronous thread mode and asynchronous request mode can be used for import. The architecture is simple, the query speed is fast, the timeliness is strong, and cold and hot layering and dynamic partitioning can be performed, which is convenient for the import and export of the real-time imported data stream. It also has good expansion and contraction performance and supports rolling upgrade.

[0096] The method for monitoring the efficiency of logistics inbound and outbound in the embodiments of the present invention has been described above. Next, the device for monitoring the efficiency of logistics inbound and outbound in the embodiments of the present invention will be described. Please refer to Figure 4 In one embodiment, the device for monitoring the efficiency of logistics inbound and outbound in the embodiments of the present invention includes:

[0097] The deduplication module 401 is configured to receive the data stream real-time input at the logistics inbound and outbound, and perform aggregation and deduplication processing on the data stream to obtain compressed data.

[0098] The import module 402 is configured to distribute and import the compressed data into a preset real-time data table using a preset asynchronous method, and update a preset offline data table using the real-time data table, where the data table stores logistics detail data, and the offline data table stores logistics index information.

[0099] The calculation module 403 is configured to calculate the inbound and outbound efficiency slices of logistics for inbound and outbound using the logistics detail data in the real-time data table, and count the inbound and outbound status information of logistics for inbound and outbound using the logistics index information in the offline data table.

[0100] The summary module 404 is configured to calculate multiple efficiency indicators for logistics inbound and outbound based on the inbound and outbound efficiency slices and the inbound and outbound status information, summarize each efficiency indicator into a materialized view and display it at the logistics inbound and outbound.

[0101] In the embodiments of the present invention, among the data sources generated during the logistics in and out of the station, there are real-time input data streams and offline data. Among them, before the real-time input data stream is imported into the real-time data table, the data stream is aggregated and deduplicated through a distributed queue to ensure that the real-time data table is obtained immediately after import, reducing the database pressure and the subsequent calculation amount of the database; the offline data is directly loaded into the offline data table to directly form the corresponding physical indicators. Through the logistics detail data in the real-time data table, the in and out efficiency slices of the physical in and out of the station can be directly calculated, and by combining the physical detail data and logistics index information in the offline data table, new physical index information can be obtained, and then the in and out status information of the logistics in and out of the station can be calculated to obtain the basic data for measuring the efficiency of the logistics in and out of the station. Finally, the in and out efficiency slices and the in and out status information are calculated for the materialized view to improve the storage timeliness and query efficiency of the in and out of the station logistics data, and at the same time improve the real-time display of the efficiency index of the logistics in and out of the station, and finally improve the overall batch processing timeliness of the data generated by the logistics in and out of the station.

[0102] Please refer to Figure 5 , another embodiment of the efficiency monitoring device for logistics in and out of the station in the embodiments of the present invention includes:

[0103] The deduplication module 401 is used to receive the data stream input in real time for the logistics in and out of the station, and perform aggregation and deduplication processing on the data stream to obtain compressed data;

[0104] The import module 402 is used to distribute and import the compressed data into a preset real-time data table by using a preset asynchronous method, and update a preset offline data table by using the real-time data table, where the data table stores logistics detail data, and the offline data table stores logistics index information;

[0105] The calculation module 403 is used to calculate the in and out efficiency slices of the logistics in and out of the station by using the logistics detail data in the real-time data table, and calculate the in and out status information of the logistics in and out of the station by using the logistics index information in the offline data table;

[0106] The summary module 404 is used to calculate multiple efficiency indicators of the logistics in and out of the station according to the in and out efficiency slices and the in and out status information, summarize each efficiency indicator into a materialized view and display it at the logistics in and out of the station.

[0107] Specifically, the efficiency monitoring device for the logistics in and out of the station further includes:

[0108] The loading module 405 is used to: obtain the offline data of the logistics in and out of the station, and load the offline data into a preset configuration table to obtain logistics configuration information; write the logistics configuration information into the preset offline data table to update the logistics index information in the offline data table.

[0109] Specifically, the deduplication module 401 includes:

[0110] An identification aggregation unit 4011, configured to extract multiple attribute identification information and corresponding attribute parameters in the data stream, and perform aggregation processing on each of the attribute identification information to obtain aggregated identification information;

[0111] A query unit 4012, configured to query a preset identification reference table by using the aggregated identification information, and determine whether the aggregated identification information exists in the identification reference table according to the query result;

[0112] A data aggregation unit 4013, configured to, if it exists, remove the corresponding attribute parameters in the aggregated identification information; if it does not exist, perform aggregation processing on the corresponding attribute parameters in the aggregated identification information to obtain compressed data.

[0113] Specifically, the import module 402 includes:

[0114] An asynchronous thread import unit 4021, configured to, if the preset asynchronous method is an asynchronous thread mode, call a corresponding number of threads from a preset thread pool according to the number of storage queues of the compressed data in a preset distributed message queue to each storage queue;

[0115] Asynchronously import the compressed data sequentially stored in each storage queue into a preset real-time data table according to the structure data storage protocol of the thread.

[0116] Specifically, the import module 402 further includes:

[0117] An asynchronous request import unit 4022, configured to, if the preset asynchronous method is an asynchronous request mode, respectively generate data storage requests by using the compressed data sequentially stored in each storage queue of the preset distributed message queue through a preset transmission protocol; asynchronously send data storage requests to a storage system where a preset real-time data table is located through each storage queue; receive response information of the storage system to each data storage request, and determine an import result of the compressed data according to the response information, where the import result includes import success and import failure.

[0118] Specifically, the summary module 404 includes:

[0119] A storage unit 4041, configured to store each of the performance indicators to a corresponding storage location in a preset materialized view framework according to the indicator type of each performance indicator to obtain a materialized view;

[0120] A display unit 4042 is configured to query, according to the interface information of the front-end interface in the logistics in-and-out station, the performance indicators mapped to the interface information in the materialized view, and obtain the corresponding performance indicators from the logistics view according to the query result for display in the front-end interface.

[0121] In the embodiments of the present invention, when calculating the performance indicators of the center in-and-out station based on real-time data and offline data, the implementation of the summary time and the base table data is unified through the materialized view. The materialized view table is automatically named during query. Compared with the storage method of the ordinary view, the calculation and query efficiency are improved. In addition, the Bitmap data is used as the basic data for calculation, analysis and statistics, which can perform accurate deduplication within a large range. Compared with other data formats, the calculation efficiency is improved, and deduplication has been achieved before storage. Then, through the data model of Doris, when compressing data and importing it into the real-time data table, the import can be carried out in an asynchronous thread mode and an asynchronous request mode. The architecture is simple, the query speed is fast, the timeliness is strong, and cold and hot layering and dynamic partitioning can be performed, which is convenient for the import and export of the real-time import data stream. It also has good expansion and contraction performance and supports rolling upgrade.

[0122] Above Figure 4 and Figure 5 The performance monitoring device for the logistics in-and-out station in the embodiments of the present invention is described in detail from the perspective of modular functional entities. Next, the performance monitoring device for the logistics in-and-out station in the embodiments of the present invention is described in detail from the perspective of hardware processing.

[0123] Figure 6 FIG. 14 is a schematic structural diagram of a performance monitoring device for a logistics in-and-out station provided by an embodiment of the present invention. The performance monitoring device 600 for the logistics in-and-out station may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) for storing application programs 633 or data 632. Among them, the memory 620 and the storage media 630 may be transient storage or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the performance monitoring device 600 for the logistics in-and-out station. Further, the processor 610 may be configured to communicate with the storage media 630 and execute a series of instruction operations in the storage media 630 on the performance monitoring device 600 for the logistics in-and-out station.

[0124] The logistics inbound and outbound efficiency monitoring device 600 may further include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, and so on. Those skilled in the art can understand that Figure 6 The structure of the logistics inbound and outbound efficiency monitoring device shown does not constitute a limitation on the logistics inbound and outbound efficiency monitoring device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0125] The present invention also provides a logistics inbound and outbound efficiency monitoring device. The computer device includes a memory and a processor. When the computer-readable instructions stored in the memory are executed by the processor, the processor is caused to execute the steps of the logistics inbound and outbound efficiency monitoring method in the above-mentioned various embodiments.

[0126] The present invention also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may also be a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a computer, the computer is caused to execute the steps of the logistics inbound and outbound efficiency monitoring method.

[0127] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, and other various media that can store program codes.

[0129] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring the efficiency of logistics in and out of a station, characterized in that, The method for monitoring the efficiency of logistics in and out of the station includes: Receiving the data stream real-time input for logistics in and out of the station, and performing aggregation and duplicate removal processing on the data stream to obtain compressed data; Using a preset asynchronous method to distributively import the compressed data into a preset real-time data table, and using the real-time data table to update a preset offline data table, where the real-time data table stores logistics detail data, and the offline data table stores logistics index information; Using the logistics detail data in the real-time data table, adopting data slicing technology, evenly distributing the obtained logistics detail data of the target type to different computing nodes, calculating the in and out efficiency slices of logistics in and out of the station through a preset algorithm, and using the logistics index information in the offline data table to count the in and out status information of logistics in and out of the station; Calculating multiple efficiency indicators of logistics in and out of the station according to the in and out efficiency slices and the in and out status information, summarizing each efficiency indicator into a materialized view and displaying it at the logistics in and out station.

2. The method for monitoring the efficiency of logistics in and out of a station according to claim 1, characterized in that, Before using the real-time data table to update the preset offline data table, it further includes: Obtaining the offline data of logistics in and out of the station, and loading the offline data into a preset configuration table to obtain logistics configuration information; Writing the logistics configuration information into the preset offline data table to update the logistics index information in the offline data table.

3. The method for monitoring the efficiency of logistics in and out of a station according to claim 1, characterized in that, The performing aggregation and duplicate removal processing on the data stream to obtain compressed data includes: Extracting multiple attribute identification information and the attribute parameters corresponding to the attribute identification information in the data stream, and performing aggregation processing on each attribute identification information to obtain aggregated identification information; Querying a preset identification reference table using the aggregated identification information, and judging whether the aggregated identification information exists in the identification reference table according to the query result; If it exists, removing the corresponding attribute parameters in the aggregated identification information; If it does not exist, performing aggregation processing on the corresponding attribute parameters in the aggregated identification information to obtain compressed data.

4. The method for monitoring the efficiency of logistics in and out of a station according to claim 1, characterized in that, The using a preset asynchronous method to distributively import the compressed data into a preset real-time data table further includes: If the preset asynchronous method is an asynchronous thread method, calling the corresponding number of threads from a preset thread pool according to the number of storage queues of the compressed data in a preset distributed message queue into each storage queue; Asynchronously importing the compressed data sequentially stored in each storage queue into the preset real-time data table according to the structure data storage protocol of the thread.

5. The method for monitoring the efficiency of logistics in and out of a station according to claim 4, characterized in that, The using a preset asynchronous method to distributively import the compressed data into a preset real-time data table further includes: If the preset asynchronous method is an asynchronous request method, respectively using the compressed data sequentially stored in each storage queue in the preset distributed message queue to generate data storage requests through a preset transmission protocol; Asynchronously sending data storage requests to the storage system where the preset real-time data table is located through each storage queue; Receiving the response information of the storage system to each data storage request, and determining the import result of the compressed data according to the response information, where the import result includes import success and import failure.

6. The method for monitoring the efficiency of logistics in and out of a station according to any one of claims 1-5, characterized in that, Summarizing each of the performance indicators into a materialized view and displaying them includes: Storing each of the performance indicators in a corresponding storage location in a preset materialized view framework according to the indicator type of each of the performance indicators to obtain a materialized view; Querying, in the materialized view, the performance indicators mapped to the interface information according to the interface information of the front-end interface in the logistics in-and-out station, and obtaining, according to the query result, the corresponding performance indicators in the materialized view for display in the front-end interface.

7. A device for monitoring the efficiency of logistics in and out of a station, characterized in that, The performance monitoring device for the logistics in-and-out station includes: A deduplication module, configured to receive the data stream real-time input in the logistics in-and-out station, and perform aggregation and deduplication processing on the data stream to obtain compressed data; An import module, configured to use a preset asynchronous method to distributively import the compressed data into a preset real-time data table, and update a preset offline data table with the real-time data table, wherein the real-time data table stores logistics detail data, and the offline data table stores logistics indicator information; A calculation module, configured to use the logistics detail data in the real-time data table, adopt a data slicing technology to evenly distribute the obtained logistics detail data of the target type to different calculation nodes, and calculate the logistics detail data through a preset algorithm to obtain the in-and-out efficiency slices of the logistics in-and-out station, and use the logistics indicator information in the offline data table to count the in-and-out status information of the logistics in-and-out station; A summarization module, configured to calculate multiple performance indicators of the logistics in-and-out station according to the in-and-out efficiency slices and the in-and-out status information, summarize each of the performance indicators into a materialized view and display it in the logistics in-and-out station.

8. The device for monitoring the efficiency of logistics in and out of a station according to claim 7, characterized in that, The performance monitoring device for the logistics in-and-out station further includes: a loading module, configured to: obtain the offline data of the logistics in-and-out station, and load the offline data into a preset configuration table to obtain logistics configuration information; write the logistics configuration information into the preset offline data table to update the logistics indicator information in the offline data table.

9. An efficiency monitoring device for logistics in and out of a station, characterized in that, The performance monitoring device for the logistics in-and-out station includes: a memory and at least one processor, and instructions are stored in the memory; The at least one processor calls the instructions in the memory to cause the performance monitoring device for the logistics in-and-out station to execute the steps of the performance monitoring method for the logistics in-and-out station according to any one of claims 1-6.

10. A computer-readable storage medium, on which instructions are stored, characterized in that, When the instructions are executed by the processor, the steps of the performance monitoring method for the logistics in-and-out station according to any one of claims 1-6 are implemented.

Images