A digital-based intelligent logistics data processing method and system

Abstract

The present invention discloses a digital-based intelligent logistics data processing method and system, which acquires logistics data, wherein the logistics data includes first logistics data within a first period and second logistics data within the first period, and is gathered in a data center, wherein the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; according to a data placement strategy, the first procurement data and the second procurement data are inserted into a procurement data set, the first warehousing data and the second warehousing data are inserted into a warehousing data set, the first distribution data and the second distribution data are inserted into a distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored independently. According to the present invention, logistics data with calculated association are uniformly placed in the same data set for storage according to the data placement strategy, thereby improving the efficiency of e-commerce logistics data processing.

Description

Technical Field

[0001] The present invention relates to the field of logistics technology, and in particular to a digital-based intelligent logistics data processing method and system. Background Art

[0002] Online shopping has become a trend. With the rapid development of online shopping, people's demand for logistics services is also increasing. Therefore, logistics services need to gradually develop from traditional manual operations to low-cost, high-efficiency, and intelligent directions. When e-commerce sells a certain product, it needs to go through various links such as procurement, warehousing, and distribution, and various types of data information are generated in this process. The current data processing technology generally records the data manually into the device and then classifies and summarizes it, which is prone to problems such as long data processing time and low efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide a digital-based intelligent logistics data processing method and system to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A digital-based intelligent logistics data processing method, comprising:

[0006] Obtaining logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and aggregating the data in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data;

[0007] The first logistics data and the second logistics data are processed, and according to the data placement strategy, the first procurement data and the second procurement data are inserted into the procurement data set, the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored separately.

[0008] Further, the first logistics data and the second logistics data are processed, and according to the data placement strategy, the first procurement data and the second procurement data are inserted into the procurement data set, the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored separately. Specifically, the steps include: according to the initial format of the data text, combining the data elements regularly;

[0009] Adjust data elements and distinguish similar data;

[0010] Divide the data into blocks, add applications to each data block, and implement association tags.

[0011] Furthermore, the information gain value of the logistics data is calculated, the data center is represented by the probability that all data information is processed, the entropy value of the data information is defined as the entropy concept value, and the entropy concept value is analyzed to determine the correlation between the entropy concept value and the logistics data information.

[0012] Furthermore, based on the application where the data resides, a storage system is selected for the application data, and the important features are used as a basis for calculating the execution time, a storage path is created for the application data, and the data is placed.

[0013] Furthermore, the execution time of each application under the storage path is calculated, and then the storage path is selected for the application before the system runs.

[0014] Furthermore, the application includes a first application, a second application and a third application that respectively map procurement, warehousing and distribution.

[0015] Furthermore, the data placement strategy is based on Hadoop strategy.

[0016] To achieve the above object, the present invention also provides the following technical solutions:

[0017] A digital-based intelligent logistics data processing system, comprising:

[0018] An acquisition module is used to acquire logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and the logistics data are aggregated in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data;

[0019] A processing module is used to process the first logistics data and the second logistics data, insert the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, insert the first warehousing data and the second warehousing data into the warehousing data set, insert the first distribution data and the second distribution data into the distribution data set, and store the procurement data set, the warehousing data set and the distribution data set independently.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention, according to the data placement strategy, uniformly places the logistics data with computational association in the same data set for storage, thereby improving the efficiency of e-commerce logistics data processing. When the logistics data is classified and stored, a suitable storage management system and device are selected for the data according to the application characteristics, and a storage path and placement mode are planned for the logistics data before operation, which can improve the rationality of storage path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the process of the present invention.

[0023] Figure 2 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] See also Figure 1 to Figure 2 , the present invention provides a technical solution:

[0026] A digital-based intelligent logistics data processing method, comprising:

[0027] Obtaining logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and aggregating the data in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data;

[0028] The first logistics data and the second logistics data are processed, and according to the data placement strategy, the first procurement data and the second procurement data are inserted into the procurement data set, the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored independently, wherein the data placement strategy is based on the Hadoop strategy.

[0029] Specifically, a product is purchased, stored and distributed (self-operated distribution or third-party distribution) by the seller as a complete cycle. The first logistics data and the second logistics data are processed, and the first procurement data and the second procurement data are inserted into the procurement data set according to the data placement strategy, and the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored separately. Specifically, it includes: according to the initial format of the data text, the data elements are combined regularly;

[0030] Adjust data elements and distinguish similar data;

[0031] Divide the data into blocks, add applications to each data block, and implement association tags.

[0032] The present invention uses a text editor to edit Hadoop strategies, combines the data elements in a regular manner in combination with the initial format of the data text; adjusts the data elements to distinguish similar data, such as distinguishing them by different colors and different numbers; divides the data blocks according to the Hadoop related mechanism, adds applications to each data block, and realizes associated marking.

[0033] Calculate the information gain value of logistics data, with the probability that all data information is processed k 1 , k 2 ,… k n Indicates data center K , K= { k 1 , k 2 ,…… k n}, where data center K The relationship with all data information is as follows:

[0034] (1);

[0035] in, k i It is expressed as the probability that the neutron data (information) in the data center is associated with all the data (information); i is the number of iterations of the data. After the calculation of formula (1), the entropy value of the data information can be defined by formula (2):

[0036] E ( k 1 , k 2 ,…… k n )= (2);

[0037] Among them, the entropy value of data information is defined as the entropy value concept value E , based on the entropy value E The concept values ​​are analyzed to determine the entropy concept value E The correlation with logistics data information. E There is a certain degree of randomness in the application. E It can be regarded as a data information evaluation indicator. E Value Analysis:

[0038] when E When it is 0, it is considered that the sub-data and stream data information are related; whenE When it is 1, it is considered that the sub-data at this time has no correlation with the stream data information. On this basis, obtain the E Values ​​within the range E The gain is calculated by using the value. It is considered that when E approaches 0 infinitely, the data has processing benefits, otherwise it is invalid data. It can be expressed as follows:

[0039] Gain( W , N )= T ( W )- (3);

[0040] in, N for{ n 1 , n 2 , n 3 ,……, n i}, Gain( W , N ) is the gain value of the entropy value concept value of logistics data information, T ( W ) is expressed as W The measurement value of logistics data information under the (information fission) state. After formula (3) is calculated, it is combined with formula (2) to make T ( W ) should meet the value requirements in the range of 1 to 0. T ( W ) satisfies the set conditions, then equation (3) can be obtained.

[0041] Specifically, according to the application where the data is located, the application includes a first application, a second application, and a third application that map procurement, warehousing, and distribution, respectively. The placement strategy setter selects the "shortest" or "optimal" storage path for the data as much as possible according to the application, storage management system, storage device type and characteristics. "Shortest" or "optimal" is measured by the length of application execution time. The shorter the selected storage path makes the application execution time, the shorter / better the path. The formal description of the shortest / optimal storage path search problem and method is as follows: Assume that the data center has l (Here l= 3) applications, namely A 1 , A 2 ,…… A k ; m storage system, denoted as S 1 ,S 2 ,…… S m ; n storage device, denoted as D 1 , D 2 ,…… D n ; First determine the application where the data is located and set it to A k (1≤ k ≤ l ), for application A k Data selection storage system S p ( 1≤ p ≤ m ) 、 D q (1≤ q ≤ n ), making the application execution time T Shortest / best, T Expressed as T ( A k , S p , D q ),in, S p and D q For application A k The shortest / optimal storage path for data in . Solving equation (4) yields S p and D q . T ( A k , S p , D q )= min ( T ( A k , S 1 , D 1 )……, T ( A k , S 1 , D n ),T ( A k , S 2 , D 1 )……, T ( A k , S 2 , D n ), T ( A k , S m , D 1 )……, T ( A k , S m , D n ))(4).

[0042] T ( A k , S p , D q ) indicates application A k The execution time of the data in the shortest / optimal storage path, min ( T ( A k , S 1 , D 1 )……, T ( A k , S 1 , D n ), T ( A k , S 2 , D 1 )……, T ( A k , S 2 , D n ), T ( A k, S m , D 1 )……, T ( A k , S m , D n )) indicates application A k The shortest / optimal execution time of the data under all storage paths. The application execution time is related to some characteristics of the application, storage management system, and storage device. The important characteristics are selected as the basis for calculating the execution time, which is expressed by formula (5).

[0043] T ( A k , S g , D h )= FA k · I a + FS g · I g + FD h · I h (5);

[0044] in, FA k , FS g , FD h Application A k , Storage Management System S g , Storage Devices D h The eigenvalue vector of I a , I g , I h are the impact factor vectors of each eigenvalue vector on the execution time. Application eigenvector FA k Including read-write ratio, read-write block size, locality strength and other factors; storage management system feature vector FS g Including read and write access latency, storage capacity, consistency strength and other factors; storage device feature vector FDh Includes factors such as device read and write access latency, throughput, and storage capacity; I a , I g , I h The application execution time can be obtained by running the load in an environment with different storage management systems and storage device combinations, and then estimated according to formula (5).

[0045] Using important features as the basis for calculating execution time, A k Data storage path, and place data. Calculate the execution time of each application under the storage path, and then select the storage path for the application before the system runs. The characteristics of the application, storage management system, and storage device are determined based on experience and parameter files. When identifying the application characteristics, each application is regarded as a whole. Once the storage path is planned for the entire application, all data generated by the application is stored according to the planned path. The execution time of each application under each storage path is calculated according to formula (5) before running, and then according to formula (4), the storage path is selected for the application before the system runs. For example, the first application, the second application, and the third application are procurement, warehousing, and distribution applications, respectively. Based on the previous understanding of the characteristics of such applications and the characteristics of various storage management systems and storage devices, the following storage path is planned for the first application: SATA SSD, as the storage point of the procurement data set; the following storage path is planned for the second application: Key-Value storage, as the storage point of the warehousing data set; the following storage path is planned for the third application: HDD, as the storage point of the distribution data set. In the first cycle, the first purchase is stored in SATA SSD, the first storage data is stored in Key-Value storage, and the first delivery data is stored in HDD. In the second cycle, the second purchase is stored in SATA SSD, the second storage data is stored in Key-Value storage, and the second delivery data is stored in HDD. In actual applications, the purchase data set, storage data set, and delivery data set are stored separately to facilitate the query of logistics data information.

[0046] like Figure 2 As shown, a digital-based intelligent logistics data processing system includes:

[0047] An acquisition module is used to acquire first logistics data within a first period and second logistics data within the first period, and aggregate them in a data center, wherein the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data;

[0048] A processing module is used to process the first logistics data and the second logistics data, insert the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, insert the first warehousing data and the second warehousing data into the warehousing data set, insert the first distribution data and the second distribution data into the distribution data set, and store the procurement data set, the warehousing data set and the distribution data set independently.

[0049] The present invention, as an optional solution: in order to reduce the data center K In i The number of iterations can save computing time and improve the efficiency of defining the entropy value of data information in (Equation (2)). mi Probability k mi hour, mi It's 1, 2, 3... n The middle value of , we need to consider the influence of the node on other adjacent nodes, so an improved probability method for the node to process the data information at the node is proposed, as shown in the following formula (6):

[0050] (6);

[0051] in, P X is the average value of all interval node probabilities; P Y is the average of all node probabilities; interval nodes such as 1 and 4, 5 and 9, etc., in order to prevent affecting the correlation, in principle, the node interval should not exceed 3, and the interval nodes start from 1 to n End, the 1st and 4th nodes have a node quantity, 1 to n The total number of nodes that exist, P i is the average value of the node quantity probability.

[0052] The present invention, the undescribed parts are prior art.

[0053] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A digital-based intelligent logistics data processing method, characterized in that: include: Obtaining logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and aggregating the data in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; The first logistics data and the second logistics data are processed, and according to the data placement strategy, the first procurement data and the second procurement data are inserted into the procurement data set, the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored independently, specifically including: combining data elements according to the initial format of the data text; adjusting the data elements to distinguish similar data; dividing the data blocks, adding applications to each data block, and realizing association tags; The entropy value of logistics data information is defined as the entropy value concept value, and the entropy value calculation formula of the logistics data information is: ； Among them, k i It represents the probability that the neutron data information in the data center is associated with all the data information; n is the total number of nodes in the data center, E is the entropy concept value, and the entropy concept value E is analyzed to determine the entropy concept value E Correlation with logistics data information: E When it is 0, it is considered that the sub-data at this time has a correlation with the logistics data information; when E When it is 1, it is considered that the sub-data at this time has no correlation with the logistics data information, and on this basis, the value in the range of 0 to 1 is obtained. E Values ​​within the range E The gain is calculated based on the value, and it is considered that when E approaches 0 infinitely, the corresponding data has processing benefits, otherwise it is invalid data; The probability k corresponding to the intermediate node mi is processed mi When mi is 1, 2, 3, ..., n The intermediate value of , using an improved probability calculation method, is as follows: ； Where P is the average probability of all interval nodes; P Y is the average value of all node probabilities; V ad The total number of nodes from 1 to n.

2. A digital-based intelligent logistics data processing method as claimed in claim 1, characterized in that: According to the application where the data is located, a storage system is selected for the application data, features are selected as the basis for calculating the execution time, a storage path is selected for the application data based on the execution time, and the data is placed.

3. A digital-based intelligent logistics data processing method as claimed in claim 2, characterized in that: Calculate the execution time of each application under the storage path and select the storage path for the application before the system runs.

4. A digital-based intelligent logistics data processing method as claimed in claim 3, characterized in that: The applications include a first application, a second application, and a third application that respectively map procurement, warehousing, and distribution.

5. The method for processing intelligent logistics data based on digitization according to claim 1, characterized in that: The data placement strategy is a Hadoop-based strategy.

6. A digital-based intelligent logistics data processing system for implementing the method according to claim 1, characterized in that: include: An acquisition module is used to acquire logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and the logistics data are aggregated in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; The processing module is used to process the first logistics data and the second logistics data, insert the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, insert the first warehousing data and the second warehousing data into the warehousing data set, insert the first distribution data and the second distribution data into the distribution data set, and store the procurement data set, the warehousing data set and the distribution data set independently, specifically including: combining the data elements according to the initial format of the data text; adjusting the data elements to distinguish similar data; dividing the data blocks, adding applications to each data block, and realizing association marking.

Images