Logistics digital management system based on AI large model

Abstract

The invention discloses a logistics digital management system based on an AI large model, which comprises a data acquisition module, a distribution model and an application end, and is characterized in that the data acquisition module is used for acquiring data of a plurality of logistics businesses generated by a plurality of nodes and uploading the acquired data to a data receiving module, and the application end is connected with a database; the reader is used for reading database data; the data receiving and acquiring module comprises a data center and an edge computing center connected with the data center, and the data center can transmit a part of logistics service data flow to the edge computing center and is deployed on the edge computing center with the minimum network bandwidth increment; the method is used for improving the transmission rate of data of logistics business from a node to a data acquisition module. The purpose of improving the transmission rate of the data of the logistics business from the node to the data acquisition module can be achieved, and then the processing efficiency of a system application end on the logistics business data can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing systems, in particular to a logistics digital management system based on an AI large model. BACKGROUND

[0002] With the rapid development of the logistics transportation industry, the business data of various logistics business parties also presents exponential growth. These data have become resources for the development of logistics enterprises, and through data statistical analysis, corresponding decisions are made to promote enterprise development.

[0003] Under the background of big data, the scale of logistics business data is growing larger and larger, and enterprises use processing systems to centrally process logistics business data. In this mode, in some application systems that require high concurrency and low latency, the above processing method will affect the performance of the application system, resulting in low processing efficiency of logistics business data. SUMMARY

[0004] The purpose of the present application is to provide a logistics digital management system based on an AI large model to solve the problems raised in the background art.

[0005] To achieve the above purpose, the present application provides the following technical solution: A logistics digital management system based on an AI large model, comprising: A data acquisition module for acquiring data of multiple logistics businesses generated by multiple nodes and uploading the acquired data to a data receiving module; An allocation model comprising a data cache area, a synchronous thread area, an asynchronous thread area, and a control group. The data receiving module is responsible for temporarily storing the uploaded data in the data cache area according to a preset strategy. The synchronous thread area and the asynchronous thread area are responsible for generating execution threads and storing the data in the data cache area in the database. The control group is responsible for receiving data and speed feedback of data storage, and for closed-loop control of the size of the data cache area and the number of threads in the thread pool; An application end connected to the database for reading database data; The data acquisition module comprises a data center and an edge computing center connected to the data center. The data center can stream a part of the logistics business data to the edge computing center, and is deployed on the edge computing center with the smallest increment of network bandwidth, to improve the transmission rate of the logistics business data from the nodes to the data acquisition module.

[0006] Further, the total bandwidth consumed by the data generated by the edge computing center does not exceed the bandwidth resources provided by the edge computing center.

[0007] Further, the preset strategy is specifically: the data cache area contains three layers of allocation areas, which are respectively front layer area, middle layer area and rear layer area, wherein the front layer area includes a plurality of linked lists, one linked list stores data attributes of data corresponding to one node, when a linked list corresponding to a node reaches the maximum length or reaches the timeout time, the data of the linked list is encapsulated into a data packet and written into the middle layer area, and after the real-time linked list in the middle layer area is full, the stored data is written into the rear layer area.

[0008] Further, the control group receives feedback information of the nodes and the database, and dynamically adjusts the size of the data cache area and the number of threads in the synchronous thread area and the asynchronous thread area according to the feedback information.

[0009] Further, the feedback information received by the control group includes the amount of data received by the synchronous thread area per unit time, the number of data storage times of the synchronous thread area per unit time, the number of times of writing into the rear layer area per unit time, the amount of data stored by the asynchronous thread area per unit time, and the database storage time of the working thread of the asynchronous thread area per unit time, the time interval of receiving data feedback by the control group is a feedback period, and the time interval of dynamically adjusting the size of the data cache area and the number of threads in the synchronous thread area and the asynchronous thread area according to the feedback is a control period.

[0010] Compared with the prior art, the beneficial effects of the present application are: The data center of the data acquisition module can transmit a part of the logistics business data stream to the edge computing center, and is deployed on the edge computing center with the minimum increment of network bandwidth, so that the application can be deployed on the node with the smallest communication overhead under the condition of meeting the bandwidth constraint, and the applications in mutual communication are deployed as close as possible to the edge node, further reducing the communication overhead. The purpose of improving the transmission rate of logistics business data from the node to the data acquisition module is achieved. Further, the processing efficiency of the application end to the logistics business data can be improved.

[0011] The present application adds a data cache layer between the data acquisition module and the data, which acts as a client of the data acquisition module, buffers and manages data for the data acquisition module, and also acts as a client of the database, responsible for sampling data storage, thereby reducing the coupling between the data acquisition module and the database, and making the data transmission or storage of the data acquisition module independent of the performance of the database. In this way, the optimal number of threads can be maintained under certain system resource conditions, and each working thread can maintain the best working state, ensuring the efficiency of task allocation and processing.

[0012] The application, through the data acquisition module preset monitoring center, comprehensively analyzes the relationship between the data of the logistics business, real-time tracks the information transfer state and abnormal information detection, realizes the system information safety collection, and ensures the stability of the overall operation of the system. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 For the logistics digital management system based on the AI large model of the application.

[0014] Figure 2 For the distribution model data cache area structure diagram of the application.

[0015] Figure 3 For the master management method flow chart in the semi-asynchronous mode of the application.

[0016] Figure 4 For the connection pool scheduling strategy flow chart of the application.

[0017] Figure 5 For the data acquisition module system structure diagram of the application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application. Embodiment 1

[0019] Please refer to Figures 1 to 4 The application provides a technical solution: A logistics digital management system based on an AI large model, comprising: A data acquisition module, which is used to acquire data of multiple pieces of logistics business generated by multiple nodes and upload the acquired data to a data receiving module; A distribution model, which comprises a data cache area, a synchronous thread area, an asynchronous thread area and a control group, the data receiving module is responsible for temporarily storing the uploaded data in the data cache area according to a preset strategy, the synchronous thread area and the asynchronous thread area are responsible for generating execution threads and storing the data in the data cache area in a database, and the control group is used to receive speed feedback of data storage and do closed-loop control on the size of the data cache area and the number of threads in the thread pool; An application end, which is connected with the database and is used to read database data. The application end is a background processing terminal, which is convenient for managers to perform relevant operations.

[0020] In this embodiment, the data of the logistics business includes transportation information data, warehouse information data, and order information data. The transportation information data refers to information data related to the transportation of goods in the logistics process, such as data such as starting location, destination, transportation tool, transportation time, transportation distance, etc. The warehouse information data refers to information data related to the storage of goods in the logistics process, such as data such as warehouse location, storage time, storage cost, etc. The order information data refers to information data related to the order of goods in the logistics process, such as data such as order number, quantity of goods, shipper, consignee, etc.

[0021] In this embodiment, the data collection of the logistics business mainly involves multiple sources, such as can include enterprise internal ERP system, WMS system, TMS system, sensor equipment (such as RFID tag and GPS positioning system), mobile phone APP and social media, etc. These data cover multiple dimensions, including order information, inventory status, transportation route, vehicle location, customer feedback, etc. Each source is regarded as a node, so the collection includes node 1, node 2… node n Since the number of nodes can be multiple, in order to reduce the redundancy of the system and simplify the system, the least data acquisition module is used to acquire data, that is, only one data acquisition module is used to acquire data for multiple nodes.

[0022] With one data acquisition module, it is necessary to deploy reasonably to make the total service delay of the related (flow) data processing application minimum under the condition of meeting the network resource constraint. In order to make the total service delay minimum, the following method can be used: The data acquisition module includes a data center and multiple edge computing centers connected with the data center, the data center can transmit a part of the logistics business data stream to the edge computing center, and is deployed on the edge computing center with the minimum increment of network bandwidth, for improving the transmission rate of the logistics business data from the node to the data acquisition module.

[0023] The total bandwidth consumed by the data generated by the edge computing center does not exceed the bandwidth resources provided by the edge computing center, that is: (1) The number of flow data processing applications to be deployed in the "data center-edge computing center" network architecture is N , that is, the set is S ={ s 1, s 1,⋯, s N}. For any flow data processing application s i , there can be multiple data producers, each data producer generates a data stream, and the set of data producers is , U j bandwidth of the edge computing center j , r q bandwidth consumed by the data (stream) q , bandwidth of the edge computing center d j data stream set within the coverage.

[0024] From a feasible solution, each time the application is re-deployed on the edge computing center that minimizes the increment of network bandwidth, so that as many applications as possible are deployed on the edge computing center to reduce the total transmission delay. To this end, the application deployment bandwidth increment Δ P ijk is defined P ijk = P t ij - P t ik bandwidth resource consumption increment s i from the original deployment location d j to the edge computing center d k , which can be represented as: (2) z t bandwidth resource of the edge computing center d t used, t time, t ∈ T .

[0025] Since the optimization goal is to minimize the total transmission delay, the bandwidth usage should be allowed to increase when the total transmission delay can be reduced. Define s i bandwidth usage efficiency Δ d j from the original deployment location d k to E ijk , which is the ratio of the increased bandwidth resource consumption and the reduced system cost, and can be represented as: (3) Δ E ijkThe greater the value, the greater the value of the cost reduction of the system per unit resource after the application is re-deployed. When there is no Δ P i1k , it indicates that the bandwidth usage of the system at this time is the smallest, at this time, select the maximum and greater than 0 Δ E ijk , will s i From d j Re-deployed on the node d k , sacrifice bandwidth resources to obtain smaller transmission delay cost.

[0026] By traversing all s i ∈ S , if there is d j ∈ D (The set of edge computing centers), so that s i Re-deployed in d j is feasible, and the optimal value of the new deployment scheme is less than the optimal value of the original deployment scheme, then select the optimal value of the optimal value d j As the local optimization of the location of the deployed application s i More communication tasks are deployed as close as possible to the edge computing center, making full use of the characteristics of the existence of a large number of small tasks in the edge computing center, without causing additional overhead.

[0027] Therefore, the data center using the data acquisition module can transmit part of the logistics business data stream to the edge computing center, and deploy on the edge computing center with the smallest network bandwidth increment, so that the application can be deployed on the node with the smallest communication overhead under the bandwidth constraint, and the applications with mutual communication are deployed as close as possible to the edge node, further reducing the communication overhead. The purpose of improving the transmission rate of logistics business data from the node to the data acquisition module is achieved.

[0028] In this embodiment, since the data acquisition module is real-time data (stream), there is a large amount of data that needs to be processed by the backend after the data acquisition module. After the data acquisition module obtains the logistics business data from the node, the data receiving module receives the logistics business data according to the preset strategy and temporarily stores the data in the data cache area of the allocation model.

[0029] In the embodiment, the allocation model adds an intermediate data cache layer between the data interface acquisition module and the data. The data cache layer acts as a client of the data interface acquisition module, buffers and manages data for the data interface acquisition module, and acts as a client of the database, responsible for sampling data storage, thereby reducing the coupling between the data interface acquisition module and the database, and making the data transmission or storage of the data interface acquisition module independent of the performance of the database.

[0030] Specifically, the preset strategy is specifically as follows: Figure 2 As shown in the figure, the data cache area includes three allocation areas, namely, a front layer area, a middle layer area, and a rear layer area. The front layer area includes a plurality of linked lists, and each linked list stores data attributes of data corresponding to a node. When a linked list corresponding to a node reaches a maximum length or a timeout time, the data of the linked list is encapsulated into a data packet and written into the middle layer area. When the real-time linked list of the middle layer area is full, the stored data is written into the rear layer area.

[0031] In the embodiment, the middle layer area is composed of a real-time buffer queue and a file buffer queue. The former stores recently written real-time data, and the latter stores cached data read from the disk cache area (i.e., the rear layer area). When a large amount of data fills the real-time linked list of the middle layer area of the allocation model, the data interface acquisition module directly writes the uploaded and stored data of the data interface acquisition module into the disk cache area, thereby avoiding data loss caused by memory overflow when a large amount of data is stored.

[0032] In the embodiment, the three data cache areas of the allocation model act as an intermediate layer between the data interface acquisition module and the database, and store data in an asynchronous manner. The data interface acquisition module can return after storing a request in the data cache area, thereby improving the concurrency of the data interface acquisition module in acquiring data of multiple nodes and reducing the coupling between the data interface acquisition module and the database. When the data storage peak is reached, the rear layer area is full, and the storage request is persisted to a disk file to ensure the safety of the data request. When the storage peak ends, the data in the disk file is read into the memory again, and is written into the database in parallel through multiple threads, thereby being able to withstand the impact of massive burst storage.

[0033] Specifically, the control group periodically receives feedback information of the nodes and the database, and dynamically adjusts the size of the data cache area and the number of threads in the synchronous thread area and the asynchronous thread area according to the feedback information, to ensure the optimal operation of the system. The feedback information received by the control group includes the amount of data received by the synchronous thread area per unit time Q r , the number of data storage times of the synchronous thread area per unit time, the number of times of writing into the rear layer area per unit time n , the amount of data stored by the asynchronous thread area per unit time DD , and the database storage time of a working thread of the asynchronous thread area per unit time tThe time interval of receiving data feedback once in the control group is a feedback period T b The time interval of making a dynamic adjustment to the data buffer size and the thread number of the synchronous thread area and the asynchronous thread area according to the feedback is a control period T c .

[0034] In this embodiment, the number of nodes in a unit time Q r and the amount of stored data Q s are the core indicators of data buffer control. The increase or decrease of the buffer is determined by comparing the indicators. The following data model is used to represent the specific indicators: Q r and Q s . (4) wherein, is the amount of data received in the i-th feedback period, and the amount of stored data is n , , is the flow difference between and .

[0035] The next control period can be determined by whether the data buffer needs to be adjusted or maintained. However, the control group of the allocation model still needs to quantify the variable memory to determine the specific value of the next control period that needs to increase or decrease the memory. If the average value of the flow difference of the next control period can be determined, the size of the memory that needs to be adjusted in the next control period can be calculated. The specific calculation is shown as follows: (5) wherein, w represents the weight value of the "past", and w ∈[0,1]; is the average value of the flow difference in the last control period.

[0036] Therefore, the possible number of node clients in the next control period is A m : A m = wA n +(1- w ) A m-1 (6) It is assumed that the feedback period is Tb , control period is T c , the next control period should adjust the memory size B size is: (7) Combining formula (5) and formula (7) can be obtained: (8) Considering the possibility of explosive data storage or database performance degradation in the next control period, the value of B size needs to be modified to further increase the memory to cope with the serious shortage of memory. The modification of B size value, the allocation model introduces a new variable D w , D w represents the number of disk files written per unit time, and the larger the value indicates that the current memory usage is more tense.

[0037] Let D ′ m represent the average number of times of writing disk in the m th control period, C m,m-1 and m represent the difference between the average number of times of writing disk in the m- th period and the th period, then: C m,m-1 = D′ m - D′ m-1 (9) (10) Therefore, the adjusted memory size is B′ size : (11) The dynamic adjustment process is as follows: Begin Input: wight1, wight2, D ′ m-1 , D ′ m-2 , Q ′ b,n-1 Output: B size Output: Thread Num for (i=0 to T c / T b ) {different = Receive Data[i]-Storge Data[i]; Sum(Client Num)+= Client Num[i]; Sum(different)+= different; Sum(D)+= DistWirte; if(|ReceiveData [i] / StorgeData[i]|>3|| (|ReceiveData [i] / StorgeData[i]|<1 / 3) EruptCount++;} Avg Different=Sum(different )* T b / T c ; Avg Clinet Num=Sum(Client Num)* T b / T c ; Threadum=wight1*Client Num[ T b / T c ]+(1-wight1)*Avg Clinet Num; if(Avg Different > 0) B size =wight2*different+(1-wight)*AvgDifferent; if(EruptCount>0.1* T b / T c ) D m =sum( D )* T b / T c ; C m,m-1 = D m - D m–1 ; C m-1,m-2 = D m-1 - D m-2 ; B size = B size *[1+ C m,m-1 / ( C m-1,m-2 + C m,m-1 )]; End.

[0038] In the embodiment, the data buffer pool working mode includes a semi-synchronous mode, and the allocation model adds a thread manager for the semi-synchronous mode, forming a semi-synchronous / semi-asynchronous mode based on management. The semi-asynchronous mode thread pool implemented by the allocation model includes four parts of a management group, a synchronous thread area, an asynchronous thread area, and a connection pool. The synchronous thread area and the asynchronous thread area are both composed of working threads, the asynchronous thread area is responsible for listening to a network port to receive messages and putting the messages into a message queue, and the synchronous thread area executes requests in the message queue in parallel. The multi-thread receiving and multi-thread storage mode of the semi-asynchronous mode can effectively avoid the performance bottleneck caused by queuing of concurrent storage requests of multiple nodes in the synchronous thread area when storing massive data, and comprehensively improves the data storage efficiency of the thread pool.

[0039] In this embodiment, the thread manager includes a dispatch 1 thread and a dispatch 2 thread, the dispatch 1 thread is responsible for synchronous thread area task scheduling, and assigns tasks to the worker threads in the synchronous thread area upon receiving a storage request. The dispatch 2 thread is responsible for asynchronous thread area task scheduling, and monitors the data cache area in real time, and assigns data to the worker threads in the asynchronous thread area for storage to the database. Through the unified scheduling of the two dispatch threads on the two thread areas, the efficiency of task allocation is ensured. The inspect thread is responsible for the health inspection of the entire thread pool, and periodically checks whether all worker threads are in a busy state for a long time, so as to clean up the dead threads in the thread pool to ensure the health of the thread pool. The master thread is responsible for the overall control of the entire thread pool, dynamically adjusts the thread pool according to the feedback of the control group, and ensures the optimal number of threads under certain system resource conditions.

[0040] In this embodiment, the master thread always waits for the adjustment command of the model controller, and after receiving the thread pool adjustment feedback of the model controller, the current thread pool status is evaluated. If the number of threads needs to be reduced, the master first finds the idle threads in the thread pool and terminates them, and if there are no idle threads, it continues to search for timeout threads in the thread pool and terminates them. The dispatch 1 and dispatch 2 threads are responsible for real-time monitoring of the network port, and if there is a data storage request, an idle worker thread is obtained and assigned a task. The inspect thread periodically audits the working time of the worker threads in the thread pool, and this audit is mainly for the worker threads in the asynchronous thread area. If a worker thread is in a busy state for a long time, the inspect thread defines it as a dead thread and performs a restart operation. The master management method in the semi-asynchronous mode is as shown in Figure 3 .

[0041] In this embodiment, the allocation model provides a database connection pool mechanism for the worker threads in the asynchronous thread area. The connection pool scheduling strategy is as shown in Figure 4 . The maximum number of connections, the minimum number of connections, and the number of added connections of the connection pool need to be initialized in the configuration file. Before performing a database operation, the worker thread obtains an idle connection from the connection pool, and after performing the database operation, the worker thread puts the connection into the connection pool, and determines whether to create a new long connection or reduce the number of long connections according to the number of active threads in the current asynchronous thread area. In this way, the optimal number of threads can be maintained under certain system resource conditions, and each worker thread can maintain the best working state, ensuring the efficiency of task allocation and processing. Embodiment 2

[0042] The original data collected is often messy and needs to be pre-processed. The pre-processing steps include data cleaning, uniform format, missing value filling, outlier processing, and data normalization. For example, missing values can be filled with mean, median or prediction model, etc.; for outliers, statistical methods (such as standard deviation method) or machine learning algorithms (such as isolation forest algorithm) are used to identify and process. The outliers are generally processed by the following formula: (12) wherein, x is the original data point, min ( x ) and max ( x ) represent the minimum and maximum values in the data set, respectively, x ′ is the normalized data point.

[0043] As shown in Figure 5 , for the logistics digital management system of the present application, the safety of the collected information is particularly important, which relates to the stability of the overall operation of the system. If there is abnormal data in the system, not only will it occupy the system memory, but it may also provide false information to the user, which will affect the relevant decision. Therefore, the data acquisition module further includes a monitoring center connected with the data center. The monitoring center combines the MRF model and the FA algorithm, and is used for comprehensive analysis of the relationship between the data of the logistics business and real-time tracking of information transfer state and abnormal information detection. The specific steps are as follows: Step 1: the order of the MRF model and the forgetting factor are represented as pp and rr , respectively, the original variance and the variance forgetting factor (i.e. forgetting coefficient, a constant) , the data to be tested x tt , tt =1,2,…,n are input into the system, wherein: (13) is the i th data point, is the mean of the data point, n- 1 is the degree of freedom.

[0044] Step 2: update the MRF model coefficient θ ={ a 1 , a 2 ,…, a p} to obtain the model prediction value : (14) is white noise error term, is the tt th measurement data. i

[0045] Step 3: Update the variance of the prediction value and calculate the membership function according to the result of step 1, which is calculated as follows: 15) TT is denoted by transpose, is the membership function of the data set B ; Step 4: Calculate the MRF model test index, which is calculated as follows: (16) Step 5: Compare the model index sum i and the model index of the information after L sum t+1 , sum t+2 , sum t+L , the number of times when the comparison result is ee ≥ ee 0 is represented as k , and ee=|sum i - sum t+i | , 0 < T < T i , L (17) Combine the values of L and k to calculate the membership function solution: (18) Step 6: Update the state transition matrix , calculate the index, and the index is represented as follows: (19) where and are the probabilities that the information is normal and abnormal, respectively, and P tt (0) = 1- P tt ​​(1). After the above operation, it can be determined whether the information in the system is normal or abnormal. If the detection result is abnormal, the data can be deleted (by the monitoring center), so that the system can be protected from being attacked by abnormal data.

[0046] The present application, the part not described is the prior art.

[0047] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely divergences, modifications, replacements and variations of the embodiments, and the scope of the application is defined by the appended claims and their equivalents.

Claims

1. A logistics digital management system based on an AI large-scale model, characterized in that, include: The data acquisition module is used to acquire data from multiple logistics transactions generated by multiple nodes and upload the acquired data to the data receiving module. The allocation model includes a data cache area, a synchronous thread area, an asynchronous thread area, and a control group. The data receiving module is responsible for temporarily storing the uploaded data in the data cache area according to a preset strategy. The synchronous thread area and the asynchronous thread area are responsible for generating execution threads and storing the data in the data cache area into the database. The control group is responsible for the speed feedback of data reception and data storage, and performs closed-loop control on the size of the data cache area and the number of threads in the thread pool. The application terminal connects to the database and is used to read database data; The data acquisition module includes a data center and an edge computing center connected to the data center. The data center can transmit a portion of the logistics business data stream to the edge computing center and is deployed on the edge computing center with the smallest network bandwidth increment, in order to improve the data transmission rate of logistics business data from nodes to the data acquisition module.

2. The logistics digital management system based on an AI large-scale model as described in claim 1, characterized in that, The total bandwidth consumed by the data generated by the edge computing center does not exceed the bandwidth resources provided by the edge computing center.

3. The logistics digital management system based on an AI large-scale model as described in claim 1, characterized in that, The preset strategy is as follows: the data cache area includes three allocation areas, namely the front layer area, the middle layer area and the back layer area. The front layer area includes several linked lists. Each linked list stores the data attributes of the data corresponding to a node. When the linked list corresponding to a node reaches its maximum length or the timeout period is reached, the data of the linked list is encapsulated into a data packet and written to the middle layer area. After the linked list in the middle layer area is full in real time, the stored data is written to the back layer area.

4. The logistics digital management system based on an AI large model as described in claim 1, characterized in that, The control group periodically receives feedback information from nodes and the database, and dynamically adjusts the size of the data buffer and the number of threads in the synchronous and asynchronous thread areas based on the feedback information.

5. The logistics digital management system based on an AI large model as described in claim 4, characterized in that, The feedback information received by the control group includes the amount of data received per unit time in the synchronous thread area, the number of times the synchronous thread area stores data per unit time, the number of times the data is written to the back layer area per unit time, the amount of data stored per unit time in the asynchronous thread area, and the single database storage time of the asynchronous thread area's worker thread. The time interval between the control group receiving one data feedback is a feedback cycle. The time interval between dynamically adjusting the size of the data buffer area and the number of threads in the synchronous and asynchronous thread areas based on the feedback is a control cycle.

Images