Logistics data processing method and device, equipment and storage medium

Abstract

The invention relates to the field of logistics, and discloses a logistics data processing method, device and equipment and a storage medium, and the method is used for carrying out data conversion, data preprocessing, dimension reduction, storage and strategic analysis on original logistics data, and generating an analysis report meeting business requirements. The method comprises the following steps: collecting original logistics data in each logistics link, and converting the original logistics data into a uniform format to obtain converted logistics data; preprocessing the converted logistics data to obtain preprocessed logistics data; carrying out dimension reduction processing on the preprocessed logistics data to obtain dimension-reduced data; establishing a data warehouse and a data view based on the dimension reduction data; and obtaining a service demand, calling a corresponding analysis strategy from a pre-constructed analysis strategy library according to the service demand, extracting dimension reduction data related to the service demand from the data warehouse based on the analysis strategy, analyzing the extracted dimension reduction data according to the analysis strategy, and generating an analysis report.

Description

Technical Field

[0001] The present invention relates to the field of logistics technology, and in particular to a logistics data processing method, device, equipment and storage medium. Background Art

[0002] With the booming and accelerating development of e-commerce, the logistics industry, as a key link in the e-commerce ecosystem, is facing unprecedented development opportunities and challenges. The processing and analysis of logistics data, as a core means to improve overall operational efficiency, reduce costs, optimize user experience and enhance market competitiveness, is becoming increasingly important. Unfortunately, the current field of logistics data processing still faces a series of problems that need to be solved urgently.

[0003] First, data connection difficulties are a major pain point. Since the logistics chain involves many participants, including e-commerce platforms, logistics companies, storage centers, distribution sites, etc., the data standards between each link are inconsistent and the system compatibility is poor, resulting in poor data circulation and difficulty in achieving efficient and real-time information sharing and collaborative operations. This not only slows down the data processing speed, but also increases the risk of data errors.

[0004] Secondly, low processing efficiency is also an issue that cannot be ignored. With the surge in logistics business volume, the amount of data generated is growing exponentially, and traditional data processing methods are often unable to cope with such a large amount of data, resulting in long data processing cycles and slow response speeds, which cannot meet the rapidly changing market needs.

[0005] Furthermore, the inaccuracy of the analysis results directly affects the scientificity and effectiveness of logistics decision-making. The complexity and multidimensionality of logistics data require a high level of data analysis capabilities, but existing technologies or algorithms have limitations in processing unstructured data, building prediction models, and detecting anomalies, which may cause the analysis results to deviate from the actual situation and fail to accurately guide the formulation and implementation of logistics strategies.

[0006] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0007] The present invention provides a logistics data processing method, device, equipment and storage medium for performing data conversion, data preprocessing, dimensionality reduction, storage and strategic analysis on raw logistics data to generate an analysis report that meets business needs.

[0008] The first aspect of the present invention provides a logistics data processing method, which includes: collecting original logistics data in each logistics link, and converting the original logistics data into a unified format to obtain converted logistics data; preprocessing the converted logistics data to obtain preprocessed logistics data; performing dimensionality reduction processing on the preprocessed logistics data to obtain reduced dimensionality data; establishing a data warehouse and data views based on the reduced dimensionality data; obtaining business needs, calling corresponding analysis strategies from a pre-built analysis strategy library according to the business needs, extracting reduced dimensionality data related to the business needs from the data warehouse based on the analysis strategies, and analyzing the extracted reduced dimensionality data according to the analysis strategies to generate an analysis report.

[0009] Optionally, in a first implementation method of the first aspect of the present invention, the raw logistics data in each logistics link is collected and converted into a unified format to obtain converted logistics data, including: confirming the data collection source to be collected, the data collection source includes an order management system, a warehouse management system, a transportation management system and a distribution management system; collecting raw logistics data from each of the data collection sources through an API interface, a database connection or a data file import method; and converting the format of the collected raw logistics data according to a preset data format standard to obtain the converted logistics data.

[0010] Optionally, in a second implementation method of the first aspect of the present invention, the preprocessing of the converted logistics data to obtain preprocessed logistics data includes: performing repeatability detection on the converted logistics data to obtain non-repetitive converted logistics data; calling a preset verification rule configuration file to perform data point value range compliance verification and data logical relationship verification on each data included in the non-repetitive converted logistics data; and setting a corresponding mark value for each data according to the results of the value range compliance verification and the data logical relationship verification to obtain preprocessed logistics data.

[0011] Optionally, in a third implementation method of the first aspect of the present invention, the pre-processed logistics data is subjected to dimensionality reduction processing to obtain reduced dimensionality data, including: based on data characteristics of the pre-processed logistics data, using a clustering algorithm to classify the pre-processed logistics data; according to the category corresponding to the pre-processed logistics data, selecting a corresponding dimensionality reduction model from a pre-built dimensionality reduction model library to perform dimensionality reduction processing on the classified pre-processed logistics data; saving the pre-processed logistics data after dimensionality reduction in a preset format to obtain reduced dimensionality data.

[0012] Optionally, in a fourth implementation method of the first aspect of the present invention, establishing a data warehouse and a data view based on the dimensionality reduction data includes: establishing a data warehouse according to the attributes of the dimensionality reduction data and the relationship between the data; importing the dimensionality reduction data into the data warehouse by batch loading; and creating a data view on the data warehouse.

[0013] Optionally, in a fifth implementation method of the first aspect of the present invention, the business requirements are obtained, corresponding analysis strategies are called from a pre-built analysis strategy library according to the business requirements, dimensionality reduction data related to the business requirements are extracted from the data warehouse based on the analysis strategies, and the extracted dimensionality reduction data are analyzed according to the analysis strategies to generate an analysis report, including: collecting historical business requirements and building an analysis strategy library according to the historical business requirements; obtaining business requirements, calling corresponding analysis strategies from the analysis strategy library according to the business requirements, parsing the analysis strategies to obtain an analysis plan, the analysis plan including a data range to be extracted and an analysis method; building a query statement based on the data range to be extracted, executing the built query statement to extract the corresponding dimensionality reduction data from the data warehouse, and analyzing the extracted dimensionality reduction data according to the analysis method to generate an analysis report.

[0014] Optionally, in a sixth implementation method of the first aspect of the present invention, the collecting of historical business needs and constructing an analysis strategy library based on the historical business needs include: collecting multiple historical business needs and corresponding analysis schemes, and determining the analysis methods and logistics data ranges corresponding to the historical business needs based on the analysis schemes; classifying the multiple historical business needs, and counting the analysis methods and logistics data ranges used by historical business needs of different categories; constructing an analysis strategy framework based on the statistical results, and creating analysis strategy entries on the analysis strategy framework to form an analysis strategy library.

[0015] The second aspect of the present invention provides a logistics data processing device, including: a conversion module, which is used to collect original logistics data in each logistics link, and convert the original logistics data into a unified format to obtain converted logistics data; a preprocessing module, which is used to preprocess the converted logistics data to obtain preprocessed logistics data; a dimensionality reduction module, which is used to perform dimensionality reduction processing on the preprocessed logistics data to obtain reduced dimensionality data; an establishment module, which is used to establish a data warehouse and data view based on the reduced dimensionality data; an analysis module, which is used to obtain business needs, call corresponding analysis strategies from a pre-built analysis strategy library according to the business needs, extract reduced dimensionality data related to the business needs from the data warehouse based on the analysis strategy, and analyze the extracted reduced dimensionality data according to the analysis strategy to generate an analysis report.

[0016] Optionally, in a first implementation method of the second aspect of the present invention, the conversion module includes: a confirmation unit, used to obtain a data collection source, the data collection source including an order management system, a warehouse management system, a transportation management system and a distribution management system; a collection unit, used to collect original logistics data from each of the data collection sources through an API interface, a database connection or a data file import method; a conversion unit, used to convert the format of the collected original logistics data according to a preset data format standard to obtain the converted logistics data.

[0017] Optionally, in a second implementation of the second aspect of the present invention, the preprocessing module includes: a detection unit, used to perform repeatability detection on the converted logistics data to obtain non-repetitive converted logistics data; a verification unit, used to call a preset verification rule configuration file to perform data point value range compliance verification and data logical relationship verification on each data included in the non-repetitive converted logistics data; a marking unit, used to set a corresponding marking value for each data according to the results of the value range compliance verification and the data logical relationship verification to obtain preprocessed logistics data.

[0018] Optionally, in a third implementation of the second aspect of the present invention, the dimensionality reduction module includes: a classification unit, used to classify the preprocessed logistics data based on the data characteristics of the preprocessed logistics data using a clustering algorithm; a dimensionality reduction unit, used to select a corresponding dimensionality reduction model from a pre-built dimensionality reduction model library according to the category corresponding to the preprocessed logistics data to perform dimensionality reduction processing on the classified preprocessed logistics data; and a saving unit, used to save the preprocessed logistics data after dimensionality reduction in a preset format to obtain dimensionality reduced data.

[0019] Optionally, in a fourth implementation of the second aspect of the present invention, the establishment module includes: a first establishment unit, which establishes a data warehouse according to the attributes of the reduced dimensionality data and the relationship between the data; an import unit, which is used to import the reduced dimensionality data into the data warehouse by batch loading; and a second establishment unit, which is used to create a data view on the data warehouse.

[0020] Optionally, in a fifth implementation of the second aspect of the present invention, the analysis module includes: a construction unit, used to collect historical business needs and construct an analysis strategy library based on the historical business needs; a parsing unit, used to obtain business needs, call corresponding analysis strategies from the analysis strategy library based on the business needs, parse the analysis strategies, and obtain analysis plans, wherein the analysis plans include a data range to be extracted and an analysis method; an analysis unit, used to construct a query statement based on the data range to be extracted, execute the constructed query statement to extract corresponding reduced dimensionality data from the data warehouse, and analyze the extracted reduced dimensionality data according to the analysis method to generate an analysis report.

[0021] The third aspect of the present invention provides a logistics data processing device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the computer-readable instructions in the memory so that the logistics data processing device executes the various steps of the logistics data processing method as described above.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the various steps of the logistics data processing method described above.

[0023] In the technical solution provided by the present invention, through a systematic process, from raw data collection to final analysis report generation, unified format conversion of data, efficient preprocessing, precise dimensionality reduction, flexible data warehouse and view construction, and analysis strategy calling and data analysis based on business needs are realized, which significantly improves the efficiency and accuracy of logistics data processing. By collecting the original logistics data in each logistics link and converting it into a unified format, the comparability and consistency of the data are ensured, and the analysis errors caused by inconsistent data formats are avoided, which provides a solid foundation for subsequent data processing and analysis. The converted logistics data is preprocessed and reduced in dimensionality, which effectively improves the data quality and reduces the complexity and redundancy of the data. Moreover, through the pre-built analysis strategy library, a rapid response to business needs is achieved. When business needs change, the appropriate analysis strategy can be quickly called from the library and the analysis plan can be parsed, thereby quickly meeting business needs and improving the market response speed and competitiveness of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A first flow chart of the logistics data processing method provided by an embodiment of the present invention;

[0025] Figure 2 A second flow chart of the logistics data processing method provided by an embodiment of the present invention;

[0026] Figure 3 A third flow chart of the logistics data processing method provided by an embodiment of the present invention;

[0027] Figure 4 A fourth flow chart of the logistics data processing method provided by an embodiment of the present invention;

[0028] Figure 5 A fifth flow chart of the logistics data processing method provided by an embodiment of the present invention;

[0029] Figure 6A sixth flow chart of the logistics data processing method provided by an embodiment of the present invention;

[0030] Figure 7 A schematic diagram of the structure of a logistics data processing device provided by an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of the structure of a logistics data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The embodiment of the present invention provides a logistics data processing method, device, equipment and storage medium, which is used to perform data conversion, data preprocessing, dimension reduction, storage and strategic analysis on raw logistics data to generate an analysis report that meets business needs. The method includes: collecting raw logistics data in each logistics link, and converting the raw logistics data into a unified format to obtain converted logistics data; preprocessing the converted logistics data to obtain preprocessed logistics data; performing dimension reduction processing on the preprocessed logistics data to obtain reduced dimension data; establishing a data warehouse and data view based on the reduced dimension data; obtaining business needs, calling the corresponding analysis strategy from the pre-built analysis strategy library according to the business needs, extracting reduced dimension data related to the business needs from the data warehouse based on the analysis strategy, and analyzing the extracted reduced dimension data according to the analysis strategy to generate an analysis report.

[0033] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A first embodiment of a logistics data processing method in an embodiment of the present invention includes:

[0035] S101. Collect original logistics data in each logistics link, and convert the original logistics data into a unified format to obtain converted logistics data.

[0036] It is understandable that the execution subject of the present invention may be a logistics data processing device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0037] In this embodiment, in each logistics link, raw logistics data is collected by using sensors, scanning devices, manual input, etc. The raw logistics data includes order data (such as order number, customer information, order status, etc.), transportation data (such as vehicle location, transportation status, route information, etc.) and delivery data (such as delivery time, delivery person information, delivery status, etc.).

[0038] In this embodiment, these raw data are converted into a unified format to facilitate subsequent processing and analysis.

[0039] S102: pre-process the converted logistics data to obtain pre-processed logistics data.

[0040] In this embodiment, preprocessing the converted logistics data includes cleaning and filling the converted logistics data and processing missing values. Cleaning the converted logistics data specifically includes removing duplicate, erroneous or invalid data.

[0041] In this embodiment, the purpose of the preprocessing step is to improve the data quality and provide a reliable data basis for subsequent analysis.

[0042] S103, performing dimensionality reduction processing on the pre-processed logistics data to obtain dimensionality reduced data.

[0043] In this embodiment, the pre-processed logistics data is subjected to dimensionality reduction processing to reduce the complexity and redundancy of the data. Dimensionality reduction methods include feature selection, principal component analysis (PCA), linear discriminant analysis (LDA), etc. Dimensionality reduction data retains the key information in the original data while reducing the complexity of data analysis.

[0044] S104. Establish a data warehouse and data view based on the dimension-reduced data.

[0045] In this embodiment, the architecture and model of the data warehouse are designed according to business needs and data characteristics, and the storage strategy, index strategy, partition strategy, etc. of the data warehouse are determined to improve query performance and data management efficiency. Then the reduced data is loaded into the data warehouse, and then data views are created on the data warehouse according to business needs and data characteristics.

[0046] In this embodiment, the data view is maintained and updated regularly to ensure that the data in the view is consistent with the data in the data warehouse.

[0047] In this embodiment, the data view is also adjusted and optimized according to changes in business requirements.

[0048] S105. Obtain business requirements, call corresponding analysis strategies from a pre-built analysis strategy library according to the business requirements, extract dimensionality reduction data related to the business requirements from the data warehouse based on the analysis strategy, analyze the extracted dimensionality reduction data according to the analysis strategy, and generate an analysis report.

[0049] In this embodiment, based on the analysis strategy, the dimension reduction data related to the business needs are extracted from the data warehouse. The analysis strategy is applied to analyze the extracted dimension reduction data, such as trend analysis, correlation analysis, prediction analysis, etc. The analysis process may involve steps such as data modeling, model training, and result verification.

[0050] In this embodiment, an analysis report is generated based on the analysis results, and the analysis report can be presented in the form of charts, tables, text descriptions, etc. The analysis report contains the purpose, method, results and conclusions of the analysis, as well as possible suggestions and improvement measures. The analysis report provides intuitive data analysis results to the business department, helping them make more informed decisions.

[0051] The present embodiment provides a logistics data processing method, which realizes unified format conversion, efficient preprocessing, precise dimension reduction, flexible data warehouse and view construction, and analysis strategy calling and data analysis based on business needs through a systematic process from raw data collection to final analysis report generation, thereby significantly improving the efficiency and accuracy of logistics data processing. By collecting raw logistics data in various logistics links and converting them into a unified format, the comparability and consistency of the data are ensured, and analysis errors caused by inconsistent data formats are avoided, providing a solid foundation for subsequent data processing and analysis. The converted logistics data is preprocessed and dimensionally reduced, which effectively improves the data quality and reduces the complexity and redundancy of the data. Moreover, through the pre-built analysis strategy library, a rapid response to business needs is achieved. When business needs change, appropriate analysis strategies can be quickly called from the library and analysis solutions can be parsed, thereby quickly meeting business needs and improving the market response speed and competitiveness of the enterprise.

[0052] See also Figure 2 The second embodiment of the logistics data processing method in the embodiment of the present invention includes:

[0053] S201. Acquire data collection sources, where the data collection sources include an order management system, a warehouse management system, a transportation management system, and a distribution management system.

[0054] In this embodiment, according to the logistics process, the data collection sources generally include an order management system (used to record order information), a warehouse management system (used to manage inventory and warehousing operations), a transportation management system (used to plan transportation routes and track transportation status), and a distribution management system (used to manage the distribution process and delivery personnel information).

[0055] S202. Collect original logistics data from various data collection sources through API interface, database connection or data file import.

[0056] In this embodiment, the appropriate collection method is selected according to the characteristics of the data collection source and the data access authority. For systems that provide API interfaces, data can be obtained through API calls; for systems that support database connections, data can be directly accessed through database query statements; for systems that can only provide data files (such as CSV, Excel, etc.), data can be loaded into a local or intermediate database through a data file import tool.

[0057] S203. According to a preset data format standard, the collected original logistics data is converted into a format to obtain converted logistics data.

[0058] In this embodiment, a unified data format standard is pre-established according to business requirements and data characteristics. The data format standard should include information such as the name of the data field, data type, data length, data format (such as date format, time format, etc.) and the relationship between data.

[0059] In this embodiment, the format of the collected raw logistics data is converted according to a pre-established data format standard, and the conversion process includes data type conversion, data encoding conversion, and data format normalization, etc. The converted logistics data has a unified data structure and format, which is convenient for subsequent data processing and analysis.

[0060] In this embodiment, by clarifying the data collection source, diversifying the collection methods and unifying the data format conversion, comprehensive, efficient and standardized collection of logistics data is achieved, providing a reliable foundation for subsequent data processing and analysis.

[0061] See also Figure 3 A third embodiment of a logistics data processing method in an embodiment of the present invention includes:

[0062] S301. Perform duplication detection on the converted logistics data to obtain non-duplicated converted logistics data.

[0063] In this embodiment, the converted logistics data is subjected to duplication detection, that is, by comparing various fields in the data set, such as order number, logistics order number, commodity information, etc., to identify and remove duplicate data records.

[0064] S302, calling a preset verification rule configuration file to perform data point value range compliance verification and data logic relationship verification on each piece of data included in the non-repetitive converted logistics data.

[0065] In this embodiment, after removing duplicate data, a preset verification rule configuration file is called to perform detailed verification on each piece of data. The preset verification rule configuration file includes various verification rules for logistics data, which are pre-defined based on business requirements and data characteristics.

[0066] Data point value domain compliance check refers to checking whether each data point in the data set is within its specified value range. For example, whether the value of the date field is within a reasonable date range, whether the value of the weight field is within a reasonable weight range, etc.

[0067] Logical relationship verification between data refers to checking whether the logical relationship between different data points in the data set is established. For example, the shipping address and the receiving address cannot be the same address (unless it is a special case such as intra-city delivery), and the change of logistics status should conform to the logical order.

[0068] S303. According to the results of the value range compliance check and the logical relationship check between the data, a corresponding tag value is set for each piece of data to obtain pre-processed logistics data.

[0069] In this embodiment, according to the verification result, the system sets a corresponding tag value for each piece of data. The tag value indicates the compliance status of the data (such as compliant / non-compliant), the severity of the verification result (such as severe / general / minor), or other custom information.

[0070] The setting of tag values ​​facilitates subsequent data processing and analysis. For example, for non-compliant data, the cause can be further investigated and corrected; for seriously non-compliant data, stricter measures may be required, such as refusing to receive or notifying relevant departments for processing.

[0071] In this embodiment, the uniqueness, accuracy and completeness of the logistics data are ensured through repeatability detection, strict verification rules and setting of mark values, providing a high-quality data foundation for subsequent data processing and analysis.

[0072] See also Figure 4 A fourth embodiment of a logistics data processing method in an embodiment of the present invention includes:

[0073] S401. Based on the data characteristics of the pre-processed logistics data, a clustering algorithm is used to classify the pre-processed logistics data.

[0074] In this embodiment, the clustering algorithm is an unsupervised learning method that can divide the samples in the data set into several categories, so that the similarity of samples in the same category is high, while the similarity of samples in different categories is low. Through the clustering algorithm, the pre-processed logistics data can be divided into several categories, and the data in each category has similar characteristics or attributes.

[0075] In this embodiment, K-means clustering is used to classify the pre-processed logistics data.

[0076] S402: According to the category corresponding to the pre-processed logistics data, a corresponding dimensionality reduction model is selected from a pre-built dimensionality reduction model library to perform dimensionality reduction processing on the classified pre-processed logistics data.

[0077] In this embodiment, the pre-built dimensionality reduction model library includes multiple dimensionality reduction models, and different dimensionality reduction models are applied to different categories of pre-processed logistics data. The dimensionality reduction models include principal component analysis model, linear discriminant analysis model, t-SNE model, singular value decomposition and non-negative matrix decomposition model.

[0078] In this embodiment, according to the category information obtained by the clustering algorithm, a dimensionality reduction model matching each category is selected from the pre-built dimensionality reduction model library. After the dimensionality reduction model is selected, the classified pre-processed logistics data is subjected to dimensionality reduction processing. The process of dimensionality reduction processing is to map high-dimensional data to low-dimensional space while retaining important information in the original data as much as possible.

[0079] S403, saving the pre-processed logistics data after dimensionality reduction in a preset format to obtain dimensionality reduction data.

[0080] In this embodiment, the pre-processed logistics data after dimensionality reduction is saved in a preset format, which may be a text file, a CSV file, a database table, etc., depending on the needs of subsequent data processing and analysis.

[0081] In this embodiment, a clustering algorithm is used to achieve reasonable classification of data, and the most appropriate dimensionality reduction model is selected for dimensionality reduction processing based on the classification results, thereby greatly reducing the dimension of the data while ensuring the validity of the data, and improving the efficiency and accuracy of data processing.

[0082] See also Figure 5 A fifth embodiment of a logistics data processing method in an embodiment of the present invention includes:

[0083] S501. Establish a data warehouse based on the attributes of the dimension-reduced data and the relationships between the data.

[0084] In this embodiment, it is first necessary to analyze the data after dimensionality reduction to obtain its attributes (such as data type, data range, etc.) and the relationship between data (such as association relationship, hierarchical structure, etc.).

[0085] Based on these analysis results, the data warehouse architecture is designed, including table structure, index strategy, partition strategy, etc., to ensure efficient storage and fast access to data.

[0086] S502: Import the dimension-reduced data into the data warehouse using batch loading.

[0087] In this embodiment, during the data loading process, selecting the batch loading method can significantly improve the efficiency of data importing.

[0088] S503. Create a data view in the data warehouse.

[0089] In this embodiment, first, the needs and purpose of the data view are obtained to determine the data content, format and style that the data view needs to display. Then, according to the results of the demand analysis, the architecture and model of the data view are designed to determine the query method, data source and data update strategy of the data view. Then, the data view is created in the data warehouse, and SQL query statements are written according to the design results or the data warehouse management tool is used to create the view. Finally, the performance of the data view is tuned, including optimizing the query statement, adjusting the view structure, etc., to improve the query speed and data display efficiency.

[0090] In this embodiment, the data view may also be performance tuned, including optimizing query statements, adjusting view structures, etc., to improve query speed and data display efficiency.

[0091] In this embodiment, a data warehouse is established based on the attributes and relationships of the dimensionality reduction data, batch loading is used to efficiently import data, and views are created on the data warehouse, thereby achieving centralized management, efficient storage and convenient access to data, and improving the efficiency and accuracy of data analysis.

[0092] See also Figure 6 A sixth embodiment of a logistics data processing method in an embodiment of the present invention includes:

[0093] S601. Collect historical business needs and build an analysis strategy library based on the historical business needs.

[0094] In this embodiment, collecting historical business needs and constructing an analysis strategy library based on the historical business needs specifically includes: collecting multiple historical business needs and corresponding analysis plans, and determining the analysis methods and logistics data ranges corresponding to the historical business needs based on the analysis plans; classifying multiple historical business needs, and counting the analysis methods and logistics data ranges used by historical business needs of different categories; constructing an analysis strategy framework based on the statistical results, and creating analysis strategy entries on the analysis strategy framework to form an analysis strategy library.

[0095] In this embodiment, the analysis strategy framework includes analysis methods and logistics data scopes for different categories of requirements.

[0096] In this embodiment, historical business needs are classified according to the nature, purpose and complexity of the needs. For example, the classification can be based on business areas (such as order processing, transportation optimization, inventory management, etc.), data types (such as time series, associated data, etc.) or analysis methods (such as predictive analysis, cluster analysis, etc.).

[0097] In this embodiment, the analysis strategy can also be iteratively optimized to improve its accuracy and applicability. In addition, an update mechanism can be established to regularly review and update the analysis strategy library.

[0098] S602: Obtain business requirements, call corresponding analysis strategies from the analysis strategy library according to the business requirements, parse the analysis strategies, and obtain analysis plans, which include the data range to be extracted and the analysis method.

[0099] In this embodiment, the corresponding analysis strategy is searched in the analysis strategy library according to the category of the business requirement.

[0100] In this embodiment, the analysis scheme includes the data range to be extracted (such as data table, field, time range, etc.) and the specific analysis method (such as statistical analysis, data mining, machine learning, etc.).

[0101] S603: construct a query statement based on the data range to be extracted, execute the constructed query statement to extract the corresponding reduced-dimensionality data from the data warehouse, and analyze the extracted reduced-dimensionality data according to the analysis method to generate an analysis report.

[0102] In this embodiment, SQL or other query languages ​​are used to construct a query statement according to the determined data range and characteristics, and then the query statement is submitted to the data warehouse to execute the query to extract the required data.

[0103] In this embodiment, according to the application of the analysis method, analysis results such as statistical charts, prediction models, association rules, etc. are generated, and according to the analysis results, a preset report template is called, and the analysis results are filled into the default parameters of the report template to obtain an analysis report. The analysis report includes parts such as data and methods, analysis results, conclusions and suggestions. The report can also include visualization elements such as charts and tables to help readers better understand and analyze the results.

[0104] In this embodiment, by pre-building an analysis strategy library, rapid response and accurate analysis of business needs are achieved, which not only improves work efficiency and accuracy, but also ensures the repeatability and scalability of the analysis process, while reducing labor costs and optimizing the use of data resources, providing enterprises with efficient, reliable, flexible and economical data analysis solutions.

[0105] The above describes the logistics data processing method in the embodiment of the present invention. The following describes the device in the embodiment of the present invention. Figure 7 , the implementation of the logistics data processing device in the embodiment of the present invention includes:

[0106] The conversion module 701 is used to collect the original logistics data in each logistics link and convert the original logistics data into a unified format to obtain converted logistics data;

[0107] A preprocessing module 702 is used to preprocess the converted logistics data to obtain preprocessed logistics data;

[0108] A dimension reduction module 703 is used to perform dimension reduction processing on the pre-processed logistics data to obtain dimension reduction data;

[0109] Establishing module 704, used to establish a data warehouse and a data view based on the dimension reduction data;

[0110] The analysis module 705 is used to obtain business requirements, call corresponding analysis strategies from a pre-built analysis strategy library according to the business requirements, extract dimensionality reduction data related to the business requirements from the data warehouse based on the analysis strategy, and analyze the extracted dimensionality reduction data according to the analysis strategy to generate an analysis report.

[0111] In this embodiment, the conversion module 701 includes: a confirmation unit 7011, which is used to obtain data collection sources, and the data collection sources include order management systems, warehouse management systems, transportation management systems and distribution management systems; a collection unit 7012, which is used to collect original logistics data from each of the data collection sources through an API interface, a database connection or a data file import method; a conversion unit 7013, which is used to convert the format of the collected original logistics data according to a preset data format standard to obtain the converted logistics data.

[0112] In this embodiment, the preprocessing module 702 includes: a detection unit 7021, which is used to perform a repeatability check on the converted logistics data to obtain non-repetitive converted logistics data; a verification unit 7022, which is used to call a preset verification rule configuration file to perform a data point value range compliance check and a data logical relationship check on each data included in the non-repetitive converted logistics data; a marking unit 7023, which is used to set a corresponding marking value for each data according to the results of the value range compliance check and the data logical relationship check to obtain preprocessed logistics data.

[0113] In this embodiment, the dimensionality reduction module 703 includes: a classification unit 7031, which is used to classify the pre-processed logistics data using a clustering algorithm based on the data characteristics of the pre-processed logistics data; a dimensionality reduction unit 7032, which is used to select a corresponding dimensionality reduction model from a pre-built dimensionality reduction model library according to the category corresponding to the pre-processed logistics data to perform dimensionality reduction processing on the classified pre-processed logistics data; a saving unit 7033, which is used to save the pre-processed logistics data after dimensionality reduction in a preset format to obtain dimensionality reduced data.

[0114] In this embodiment, the establishment module 704 includes: a first establishment unit 7041, which establishes a data warehouse according to the attributes of the reduced dimension data and the relationship between the data; an import unit 7042, which is used to import the reduced dimension data into the data warehouse by batch loading; and a second establishment unit 7043, which is used to create a data view on the data warehouse.

[0115] In this embodiment, the analysis module 705 includes: a construction unit 7051, which is used to collect historical business needs and construct an analysis strategy library based on the historical business needs; a parsing unit 7052, which is used to obtain business needs, call corresponding analysis strategies from the analysis strategy library based on the business needs, parse the analysis strategies, and obtain analysis plans, which include the data range to be extracted and the analysis method; an analysis unit 7053, which is used to construct a query statement based on the data range to be extracted, execute the constructed query statement to extract the corresponding reduced dimensionality data from the data warehouse, and analyze the extracted reduced dimensionality data according to the analysis method to generate an analysis report.

[0116] In this embodiment, through a systematic process, from raw data collection to final analysis report generation, unified data format conversion, efficient preprocessing, precise dimensionality reduction, flexible data warehouse and view construction, and analysis strategy calling and data analysis based on business needs are realized, which significantly improves the efficiency and accuracy of logistics data processing. By collecting the original logistics data in each logistics link and converting it into a unified format, the comparability and consistency of the data are ensured, and the analysis errors caused by inconsistent data formats are avoided, which provides a solid foundation for subsequent data processing and analysis. The converted logistics data is preprocessed and reduced in dimensionality, which effectively improves the data quality and reduces the complexity and redundancy of the data. Moreover, through the pre-built analysis strategy library, a rapid response to business needs is achieved. When business needs change, the appropriate analysis strategy can be quickly called from the library and the analysis plan can be parsed, thereby quickly meeting business needs and improving the company's market response speed and competitiveness.

[0117] Figure 7 The structure of the logistics data processing device shown does not constitute a limitation on the logistics data processing device, and can implement the steps of the logistics data processing method provided by the above-mentioned method embodiments.

[0118] above Figure 7 The logistics data processing device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the logistics data processing equipment in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0119] Figure 8 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 810 (for example, one or more processors) and a memory 820, and one or more storage media 830 (for example, one or more mass storage devices) storing application programs 833 or data 832. Among them, the memory 820 and the storage medium 830 may be short-term storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown), and each module may include a series of instruction operations in the device 800. Furthermore, the processor 810 may be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium on the device 800.

[0120] The device 800 may also include one or more power supplies 840, one or more wired or wireless network interfaces 850, one or more input and output interfaces 860, and / or one or more operating systems 831, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc.

[0121] An embodiment of the present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the logistics data processing method.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0124] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A logistics data processing method, characterized in that: The logistics data processing method comprises: Collecting raw logistics data in each logistics link, and converting the raw logistics data into a unified format to obtain converted logistics data; Preprocessing the converted logistics data to obtain preprocessed logistics data; Performing dimensionality reduction processing on the pre-processed logistics data to obtain dimensionality reduction data; Establishing a data warehouse and a data view based on the dimension-reduced data; Obtain business requirements, call corresponding analysis strategies from a pre-built analysis strategy library according to the business requirements, extract dimensionality reduction data related to the business requirements from the data warehouse based on the analysis strategy, analyze the extracted dimensionality reduction data according to the analysis strategy, and generate an analysis report.

2. The logistics data processing method according to claim 1, characterized in that: The collecting of the original logistics data in each logistics link and converting the original logistics data into a unified format to obtain converted logistics data includes: Confirming the data collection sources to be collected, the data collection sources include order management system, warehouse management system, transportation management system and distribution management system; Collecting raw logistics data from each of the data collection sources through an API interface, database connection or data file import; According to the preset data format standard, the collected original logistics data is format converted to obtain the converted logistics data.

3. The logistics data processing method according to claim 1, characterized in that: The preprocessing of the converted logistics data to obtain preprocessed logistics data includes: Perform duplication detection on the converted logistics data to obtain non-duplicate converted logistics data; Call the preset verification rule configuration file to perform data point value range compliance verification and data logic relationship verification on each piece of data included in the non-repetitive converted logistics data; According to the results of the data point value domain compliance check and the logical relationship check between data, a corresponding tag value is set for each piece of data to obtain the pre-processed logistics data.

4. The logistics data processing method according to claim 1, characterized in that: The performing dimensionality reduction processing on the pre-processed logistics data to obtain dimensionality reduction data includes: Based on the data characteristics of the pre-processed logistics data, the pre-processed logistics data is classified using a clustering algorithm; According to the categories corresponding to the pre-processed logistics data, a corresponding dimensionality reduction model is selected from a pre-built dimensionality reduction model library to perform dimensionality reduction processing on the classified pre-processed logistics data; The pre-processed logistics data after dimensionality reduction is saved in a preset format to obtain dimensionality reduced data.

5. The logistics data processing method according to claim 1, characterized in that: The establishing of a data warehouse and a data view based on the dimension reduction data includes: Establishing a data warehouse according to the attributes of the dimension-reduced data and the relationship between the data; Importing the dimension-reduced data into the data warehouse by batch loading; A data view is created on the data warehouse.

6. The logistics data processing method according to claim 1, characterized in that: The obtaining of business requirements, calling corresponding analysis strategies from a pre-built analysis strategy library according to the business requirements, extracting dimension reduction data related to the business requirements from the data warehouse based on the analysis strategies, and analyzing the extracted dimension reduction data according to the analysis strategies to generate an analysis report, including: Collect historical business needs and build an analysis strategy library based on the historical business needs; Obtaining business requirements, calling corresponding analysis strategies from the analysis strategy library according to the business requirements, parsing the analysis strategies, and obtaining analysis plans, wherein the analysis plans include the data range to be extracted and the analysis methods; A query statement is constructed based on the data range to be extracted, the constructed query statement is executed to extract the corresponding reduced-dimensionality data from the data warehouse, and the extracted reduced-dimensionality data is analyzed according to the analysis method to generate an analysis report.

7. The logistics data processing method according to claim 6, characterized in that: The collecting of historical business requirements and building an analysis strategy library according to the historical business requirements include: Collect multiple historical business needs and corresponding analysis plans, and determine the analysis method and logistics data range corresponding to the historical business needs according to the analysis plan; Classify the plurality of historical business demands, and count the analysis methods and logistics data ranges used by the historical business demands of different categories; Based on the statistical results, an analysis strategy framework is constructed, and analysis strategy entries are created on the analysis strategy framework to form an analysis strategy library.

8. A logistics data processing device, characterized in that: include: A conversion module is used to collect original logistics data in each logistics link and convert the original logistics data into a unified format to obtain converted logistics data; A preprocessing module, used for preprocessing the converted logistics data to obtain preprocessed logistics data; A dimension reduction module, used for performing dimension reduction processing on the pre-processed logistics data to obtain dimension reduction data; Establishing a module, used for establishing a data warehouse and a data view based on the dimension reduction data; The analysis module is used to obtain business requirements, call corresponding analysis strategies from a pre-built analysis strategy library according to the business requirements, extract dimensionality reduction data related to the business requirements from the data warehouse based on the analysis strategy, and analyze the extracted dimensionality reduction data according to the analysis strategy to generate an analysis report.

9. A logistics data processing device, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the various steps of the logistics data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the various steps of the logistics data processing method as described in any one of claims 1 to 7 are implemented.

Images