Intelligent logistics data processing method and system based on digitization

Abstract

The invention discloses a digitization-based intelligent logistics data processing method and system, and the method comprises the steps: obtaining logistics data, enabling the logistics data to comprise first logistics data in a first period and second logistics data in the first period, enabling the first logistics data to be converged in a data center, enabling the first logistics data to comprise first purchasing, warehousing and distribution data, and enabling the second logistics data to comprise second purchasing, warehousing and distribution data; the second logistics data comprises second purchase, storage and distribution data; inserting the first purchase data and the second purchase data into a purchase data set according to a data placement strategy, inserting the first storage data and the second storage data into a storage data set, and inserting the first distribution data and the second distribution data into a distribution data set, and independently storing the purchase data set, the storage data set and the distribution data set. According to the method and the device, the logistics data of which the relevance is calculated are uniformly placed in the same data set for storage according to the data placement strategy, so that the processing efficiency of the e-commerce logistics data is improved.

Description

Technical Field

[0001] The present invention relates to the field of logistics technology, and in particular to a digital-based intelligent logistics data processing method and system. Background Art

[0002] Online shopping has become a trend. With the rapid development of online shopping, people's demand for logistics services is also increasing. Therefore, logistics services need to gradually develop from traditional manual operations to low-cost, high-efficiency, and intelligent directions. When e-commerce sells a certain product, it needs to go through various links such as procurement, warehousing, and distribution, and various types of data information are generated in this process. The current data processing technology generally records the data manually into the device and then classifies and summarizes it, which is prone to problems such as long data processing time and low efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide a digital-based intelligent logistics data processing method and system to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions: A digital-based intelligent logistics data processing method, comprising: Obtaining logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and aggregating the data in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; The first logistics data and the second logistics data are processed, and according to the data placement strategy, the first procurement data and the second procurement data are inserted into the procurement data set, the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored separately.

[0005] Further, the first logistics data and the second logistics data are processed, and according to the data placement strategy, the first procurement data and the second procurement data are inserted into the procurement data set, the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored separately. Specifically, the steps include: according to the initial format of the data text, combining the data elements regularly; Adjust data elements and distinguish similar data; Divide the data into blocks, add applications to each data block, and implement association tags.

[0006] Furthermore, the information gain value of the logistics data is calculated, the data center is represented by the probability that all data information is processed, the entropy value of the data information is defined as the entropy concept value, and the entropy concept value is analyzed to determine the correlation between the entropy concept value and the logistics data information.

[0007] Furthermore, based on the application where the data resides, a storage system is selected for the application data, and the important features are used as a basis for calculating the execution time, a storage path is created for the application data, and the data is placed.

[0008] Furthermore, the execution time of each application under the storage path is calculated, and then the storage path is selected for the application before the system runs.

[0009] Furthermore, the application includes a first application, a second application and a third application that respectively map procurement, warehousing and distribution.

[0010] Furthermore, the data placement strategy is based on Hadoop strategy.

[0011] To achieve the above object, the present invention also provides the following technical solutions: A digital-based intelligent logistics data processing system, comprising: An acquisition module is used to acquire logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and the logistics data are aggregated in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; A processing module is used to process the first logistics data and the second logistics data, insert the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, insert the first warehousing data and the second warehousing data into the warehousing data set, insert the first distribution data and the second distribution data into the distribution data set, and store the procurement data set, the warehousing data set and the distribution data set independently.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The present invention, according to the data placement strategy, uniformly places the logistics data with computational association in the same data set for storage, thereby improving the efficiency of e-commerce logistics data processing. When the logistics data is classified and stored, a suitable storage management system and device are selected for the data according to the application characteristics, and a storage path and placement mode are planned for the logistics data before operation, which can improve the rationality of storage path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the process of the present invention.

[0014] Figure 2 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] See also Figure 1 to Figure 2 , the present invention provides a technical solution: A digital-based intelligent logistics data processing method, comprising: Obtaining logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and aggregating the data in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; The first logistics data and the second logistics data are processed, and according to the data placement strategy, the first procurement data and the second procurement data are inserted into the procurement data set, the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored independently, wherein the data placement strategy is based on the Hadoop strategy.

[0017] Specifically, a product is purchased, stored and distributed (self-operated distribution or third-party distribution) by the seller as a complete cycle. The first logistics data and the second logistics data are processed, and the first procurement data and the second procurement data are inserted into the procurement data set according to the data placement strategy, and the first warehousing data and the second warehousing data are inserted into the warehousing data set, and the first distribution data and the second distribution data are inserted into the distribution data set, and the procurement data set, the warehousing data set and the distribution data set are stored separately. Specifically, it includes: according to the initial format of the data text, the data elements are combined regularly;

[0018] Adjust data elements and distinguish similar data; Divide the data into blocks, add applications to each data block, and implement association tags.

[0019] The present invention uses a text editor to edit Hadoop strategies, combines the data elements in a regular manner in combination with the initial format of the data text; adjusts the data elements to distinguish similar data, such as distinguishing them by different colors and different numbers; divides the data blocks according to the Hadoop related mechanism, adds applications to each data block, and realizes associated marking.

[0020] Calculate the information gain value of logistics data, with the probability that all data information is processed k 1 , k 2 ,… k n Indicates data center K , K= { k 1 , k 2 ,…… k n}, where data center K The relationship with all data information is as follows: (1); in, k i It is expressed as the probability that the neutron data (information) in the data center is associated with all the data (information); i is the number of iterations of the data. After the calculation of formula (1), the entropy value of the data information can be defined by formula (2):

[0021] E ( k 1 , k 2 ,…… k n )= (2); Among them, the entropy value of data information is defined as the entropy value concept value E , based on the entropy value E The concept values ​​are analyzed to determine the entropy concept value E The correlation with logistics data information. E There is a certain degree of randomness in the application. E It can be regarded as a data information evaluation indicator. E Value Analysis:

[0022] when E When it is 0, it is considered that the sub-data and stream data information are related; when E When it is 1, it is considered that the sub-data at this time has no correlation with the stream data information. On this basis, obtain the E Values ​​within the range E The gain is calculated by using the value. It is considered that when E approaches 0 infinitely, the data has processing benefits, otherwise it is invalid data. It can be expressed as follows:

[0023] Gain( W , N )=T ( W )- (3); in, N for{ n 1 , n 2 , n 3 ,……, n i}, Gain( W , N ) is the gain value of the entropy concept value of logistics data information, T ( W ) is expressed as W The measurement value of logistics data information under the (information fission) state. After formula (3) is calculated, it is combined with formula (2) to make T ( W ) should meet the value requirements in the range of 1 to 0. T ( W ) satisfies the set conditions, then equation (3) can be obtained.

[0024] Specifically, according to the application where the data is located, the application includes a first application, a second application, and a third application that map procurement, warehousing, and distribution, respectively. The placement strategy setter selects the "shortest" or "optimal" storage path for the data as much as possible according to the application, storage management system, storage device type and characteristics. "Shortest" or "optimal" is measured by the length of application execution time. The shorter the selected storage path makes the application execution time, the shorter / better the path. The formal description of the shortest / optimal storage path search problem and method is as follows: Assume that the data center has l (Here l= 3) applications, namely A 1 , A 2 ,…… A k ; m storage system, denoted as S 1 , S 2 ,…… S m ; n storage device, denoted as D 1 , D 2 ,…… D n ; First determine the application where the data is located and set it to A k (1≤ k≤ l ), for application A k Data selection storage system S p ( 1≤ p ≤ m ) 、 D q (1≤ q ≤ n ), making the application execution time T Shortest / best, T Expressed as T ( A k , S p , D q ),in, S p and D q For application A k The shortest / optimal storage path for data in . Solving equation (4) yields S p and D q . T ( A k , S p , D q )= min ( T ( A k , S 1 , D 1 )……, T ( A k , S 1 , D n ), T ( A k , S 2 , D 1 )……, T ( A k , S 2 , D n ), T ( Ak , S m , D 1 )……, T ( A k , S m , D n ))(4).

[0025] T ( A k , S p , D q ) indicates application A k The execution time of the data in the shortest / optimal storage path, min ( T ( A k , S 1 , D 1 )……, T ( A k , S 1 , D n ), T ( A k , S 2 , D 1 )……, T ( A k , S 2 , D n ), T ( A k , S m , D 1 )……, T ( A k , S m , D n )) indicates application A kThe shortest / optimal execution time of the data under all storage paths. The application execution time is related to some characteristics of the application, storage management system, and storage device. The important characteristics are selected as the basis for calculating the execution time, which is expressed by formula (5).

[0026] T ( A k , S g , D h )= FA k · I a + FS g · I g + FD h · I h (5); in, FA k , FS g , FD h Application A k , Storage Management System S g , Storage Devices D h The eigenvalue vector of I a , I g , I h are the impact factor vectors of each eigenvalue vector on the execution time. Application eigenvector FA k Including read-write ratio, read-write block size, locality strength and other factors; storage management system feature vector FS g Including read and write access latency, storage capacity, consistency strength and other factors; storage device feature vector FD h Includes factors such as device read and write access latency, throughput, and storage capacity; I a , I g , I h The application execution time can be obtained by running the load in an environment with different storage management systems and storage device combinations, and then estimated according to formula (5).

[0027] Using important features as the basis for calculating execution time, A k Data storage path, and place data. Calculate the execution time of each application under the storage path, and then select the storage path for the application before the system runs. The characteristics of the application, storage management system, and storage device are determined based on experience and parameter files. When identifying the application characteristics, each application is regarded as a whole. Once the storage path is planned for the entire application, all data generated by the application is stored according to the planned path. The execution time of each application under each storage path is calculated according to formula (5) before running, and then according to formula (4), the storage path is selected for the application before the system runs. For example, the first application, the second application, and the third application are procurement, warehousing, and distribution applications, respectively. Based on the previous understanding of the characteristics of such applications and the characteristics of various storage management systems and storage devices, the following storage path is planned for the first application: SATA SSD, as the storage point of the procurement data set; the following storage path is planned for the second application: Key-Value storage, as the storage point of the warehousing data set; the following storage path is planned for the third application: HDD, as the storage point of the distribution data set. In the first cycle, the first purchase is stored in SATA SSD, the first storage data is stored in Key-Value storage, and the first delivery data is stored in HDD. In the second cycle, the second purchase is stored in SATA SSD, the second storage data is stored in Key-Value storage, and the second delivery data is stored in HDD. In actual applications, the purchase data set, storage data set, and delivery data set are stored separately to facilitate the query of logistics data information.

[0028] like Figure 2 As shown, a digital-based intelligent logistics data processing system includes: An acquisition module is used to acquire first logistics data within a first period and second logistics data within the first period, and aggregate them in a data center, wherein the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; A processing module is used to process the first logistics data and the second logistics data, insert the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, insert the first warehousing data and the second warehousing data into the warehousing data set, insert the first distribution data and the second distribution data into the distribution data set, and store the procurement data set, the warehousing data set and the distribution data set independently.

[0029] The present invention, as an optional solution: in order to reduce the data center K In iThe number of iterations can save computing time and improve the efficiency of defining the entropy value of data information in (Equation (2)). mi Probability k mi hour, mi It's 1, 2, 3... n The middle value of , we need to consider the influence of the node on other adjacent nodes, so an improved probability method for the node to process the data information at the node is proposed, as shown in the following formula (6): (6); in, P X is the average value of all interval node probabilities; P Y is the average of all node probabilities; interval nodes such as 1 and 4, 5 and 9, etc., in order to prevent affecting the correlation, in principle, the node interval should not exceed 3, and the interval nodes start from 1 to n End, the 1st and 4th nodes have a node quantity, 1 to n The total number of nodes that exist, P i is the average value of the node quantity probability.

[0030] The present invention, the undescribed parts are prior art.

[0031] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A digital-based intelligent logistics data processing method, characterized in that: include: Obtaining logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and aggregating the data in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; Processing the first logistics data and the second logistics data, inserting the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, inserting the first warehousing data and the second warehousing data into the warehousing data set, inserting the first distribution data and the second distribution data into the distribution data set, and independently storing the procurement data set, the warehousing data set, and the distribution data set; Processing the first logistics data and the second logistics data, inserting the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, inserting the first warehousing data and the second warehousing data into the warehousing data set, inserting the first distribution data and the second distribution data into the distribution data set, and independently storing the procurement data set, the warehousing data set and the distribution data set, specifically including: combining the data elements according to the initial format of the data text; Adjust data elements and distinguish similar data; Divide the data into blocks, add applications to each data block, and implement association tags.

2. A digital-based intelligent logistics data processing method as claimed in claim 1, characterized in that: The information gain value of logistics data is calculated, and the data center is represented by the probability that all data information is processed. The entropy value of the data information is defined as the entropy concept value, and the entropy concept value is analyzed to determine the correlation between the entropy concept value and the logistics data information.

3. The method for processing digitalized intelligent logistics data according to claim 1, characterized in that: Select a storage system for the application data based on the application where the data is located, use important features as the basis for calculating the execution time, select a storage path for the application data based on the execution time, and place the data.

4. A digital-based intelligent logistics data processing method as claimed in claim 3, characterized in that: Calculate the execution time of each application under the storage path, and then select the storage path for the application before the system runs.

5. A digital-based intelligent logistics data processing method as claimed in claim 4, characterized in that: The applications include a first application, a second application, and a third application that respectively map procurement, warehousing, and distribution.

6. A digital-based intelligent logistics data processing method as claimed in claim 1, characterized in that: The data placement strategy is based on Hadoop strategy.

7. A digital-based intelligent logistics data processing system, characterized in that: include: An acquisition module is used to acquire logistics data, the logistics data includes first logistics data within a first period and second logistics data within the first period, and the logistics data are aggregated in a data center, the first logistics data includes first procurement, warehousing and distribution data, and the second logistics data includes second procurement, warehousing and distribution data; a processing module, configured to process the first logistics data and the second logistics data, insert the first purchase data and the second purchase data into the purchase data set according to the data placement strategy, insert the first warehousing data and the second warehousing data into the warehousing data set, insert the first distribution data and the second distribution data into the distribution data set, and store the purchase data set, the warehousing data set, and the distribution data set independently; Processing the first logistics data and the second logistics data, inserting the first procurement data and the second procurement data into the procurement data set according to the data placement strategy, inserting the first warehousing data and the second warehousing data into the warehousing data set, inserting the first distribution data and the second distribution data into the distribution data set, and independently storing the procurement data set, the warehousing data set and the distribution data set, specifically including: combining the data elements according to the initial format of the data text; Adjust data elements and distinguish similar data; Divide the data into blocks, add applications to each data block, and implement association tags.

Images