Cold chain sorting optimization system and method based on digital twinning and reinforcement learning

Abstract

The invention discloses a cold chain sorting optimization system and method based on digital twinning and reinforcement learning, and belongs to intelligent logistics equipment control, and the method comprises the steps: obtaining target object point cloud and environment temperature data, and generating a feature manifold state quantity through coding mapping; executing action sampling in a simulation environment based on the characteristic manifold state quantity, and obtaining a reference control parameter through shearing iteration; calculating the cosine similarity between the migration features and the historical feature set to select an initialization parameter, and executing gradient updating to obtain a migration control parameter; a real-time action sequence is collected, information entropy is calculated, and optimization control parameters are generated when the uncertainty boundary is broken through; and calculating a consistency error between simulation and real-time action, and correcting the reference control parameters and updating the historical feature set when a dynamic error threshold is broken through. According to the method, a virtual-real migration and dynamic entropy weight feedback mechanism is adopted, online self-adaptive correction of a control strategy can be achieved, and the sorting precision and robustness of the robot in the cold chain environment are improved.

Description

Technical Field

[0001] This invention relates to intelligent logistics equipment control, and in particular to a cold chain sorting optimization system and method based on digital twins and reinforcement learning. Background Technology

[0002] Cold chain warehouse operations typically involve handling, turnover, and picking / sorting goods in low-temperature environments. To improve warehouse operational efficiency and reduce the continuous workload of manual labor under low-temperature conditions, related technologies are gradually incorporating unmanned equipment and intelligent scheduling and control solutions. Examples include task allocation and path planning for unmanned forklifts within warehouses, which falls under the technical field of unmanned forklift scheduling. Furthermore, cold chain warehouses generally exhibit characteristics such as dense shelving, narrow aisles, and prolonged low temperatures, and may also contain dynamic obstacles such as personnel and equipment, posing requirements for safety and precise positioning and navigation during warehouse operations.

[0003] In related technologies, Chinese patent CN120235559A discloses a method and system for dynamic task scheduling of unmanned forklifts based on deep reinforcement learning in cold chain warehouses. The method involves deploying sensors in the target cold chain warehouse to collect environmental data, constructing a digital twin model of the target cold chain warehouse in the cloud based on digital twin technology, building a dynamic environmental state of the target cold chain warehouse based on sensor feedback data, generating an initial unmanned forklift trajectory based on a path generation algorithm, and optimizing the initial unmanned forklift trajectory using a multi-feature coupled spatiotemporal joint path planning algorithm deployed in the cloud to generate the optimal path and scheduling plan. The cloud then sends instructions to the edge gateway, which decomposes the instructions and sends them to each unmanned forklift.

[0004] Regarding the aforementioned technologies, the inventors believe that in the actual operation of cold chain warehouses, existing technologies still generally have shortcomings in dynamic task scheduling: First, it is difficult to achieve timely real-time response when facing dynamic changes in the warehouse environment; second, it is difficult to fully take into account the comprehensive needs of multiple optimization objectives; third, under conditions of dense shelving, narrow aisles, and the coexistence of obstacles, it is still difficult to reliably meet the requirements for accurate positioning and navigation as well as safe operation. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a cold chain sorting optimization system and method based on digital twins and reinforcement learning. Employing a virtual-real migration and dynamic entropy weight feedback mechanism, it enables online adaptive correction of control strategies, thereby improving the sorting accuracy and robustness of robots in cold chain environments.

[0006] The above objectives can be achieved through the following approach: A cold chain sorting optimization system and method based on digital twins and reinforcement learning includes: a point cloud temperature feature encoding module for acquiring point cloud data of the target object and ambient temperature data; extracting geometric and material property features through model encoding to construct a multidimensional feature vector; mapping the multidimensional feature vector to a vector space to generate a feature manifold state variable; a digital twin policy iteration module for initializing the digital twin simulation environment; performing action sampling based on the feature manifold state variable to obtain a simulation state action sequence; calculating the probability ratio between the current policy distribution and the sampled policy distribution; and obtaining baseline control parameters through shearing iteration; and a migration loss gradient calculation module for extracting migration multidimensional feature vectors and migration state action sequences from the target object; calculating the cosine similarity between the migration multidimensional feature vectors and a preset historical multidimensional feature vector set; selecting the baseline control parameter corresponding to the maximum value as the initialization parameter. The loss function gradient is calculated for the migration state action sequence, and gradient updates are performed on the initialization parameters to obtain migration control parameters. An online control parameter update module is used to collect real-time state action sequences during physical execution, calculate the information entropy sequence of the policy distribution, and construct an uncertainty boundary by calculating the mean and standard deviation of the information entropy sequence within a sliding window. When the information entropy sequence exceeds the uncertainty boundary, gradient updates are performed on the migration control parameters to generate optimized control parameters. A consistency error calculation module is used to generate updated simulation state action sequences using the optimized control parameters, calculate the consistency error with the real-time state action sequence, calculate a dynamic error threshold using a sliding window, and perform a shearing iteration to correct the baseline control parameters when the consistency error exceeds the dynamic error threshold. The multidimensional feature vector is then recalculated and written into the historical multidimensional feature vector set.

[0007] Optionally, the point cloud temperature feature encoding module includes: a point cloud parsing and temperature alignment unit, used to parse the three-dimensional spatial coordinates and normal vector data in the point cloud data, and simultaneously extract the ambient temperature time series data aligned with the timestamp of the point cloud data; a geometric and material feature encoding unit, used to perform volume integration and surface area calculation on the three-dimensional spatial coordinates to generate geometric features, and obtain the elastic modulus parameter and friction coefficient parameter of the target object to generate material property features; and a vector space mapping unit, used to perform channel concatenation on the geometric features, the material property features and the ambient temperature time series data to construct a multi-dimensional feature vector, input the multi-dimensional feature vector to a preset encoder network, perform nonlinear feature mapping, and output the feature manifold state quantity.

[0008] Optionally, the digital twin strategy iteration module includes: a simulation sampling execution unit, used to load the target object in the digital twin simulation environment, calculate the probability distribution of the grasping action based on the characteristic manifold state quantity, and perform random selection according to the probability distribution to generate a simulation state action sequence; a value deviation calculation unit, used to calculate the weighted sum of the immediate reward value and the next state prediction value through the simulation state action sequence, and calculate the difference between the weighted sum and the current state prediction value to generate a temporal difference error; and a shearing update unit, used to calculate the probability ratio of the same action in the current update process and the sampling process, perform a shearing constraint operation on the probability ratio, and perform gradient ascent calculation with the temporal difference error to output a baseline control parameter.

[0009] Optionally, the output baseline control parameters include: constructing a numerical clipping range centered on a unit value of 1; determining whether the probability ratio falls within the numerical clipping range; when the probability ratio exceeds the numerical clipping range, mapping the probability ratio to the boundary value of the numerical clipping range to generate a truncated ratio sequence; calculating the first product of the probability ratio and the time series difference error, and the second product of the truncated ratio sequence and the time series difference error, respectively; selecting the minimum value between the first product and the second product to construct an objective function; and performing gradient updates on the initialization parameters to output the baseline control parameters.

[0010] Optionally, the migration loss gradient calculation module includes: a feature similarity optimization subunit, used to perform a dot product operation on each vector in the migration multidimensional feature vector and the historical multidimensional feature vector set, and divide by the product of the vector magnitudes to obtain a similarity value sequence, extract the maximum similarity value from the similarity value sequence, and use the benchmark control parameter of the maximum similarity value as an initialization parameter; an action residual gradient calculation subunit, used to calculate the numerical deviation between the predicted action distribution and the actual action distribution using the migration state action sequence and the initialization parameter, and generate a parameter gradient vector based on the numerical deviation through backpropagation; and a parameter migration correction subunit, used to calculate the product of the parameter gradient vector and the maximum similarity value to obtain a correction amount, perform a vector difference operation on the initialization parameter and the correction amount, and output migration control parameters.

[0011] Optionally, the method further includes: extracting the Euclidean norm of the characteristic manifold state variables, calculating the absolute value of the temporal difference error, multiplying the Euclidean norm and the absolute value to generate a state sensitivity value; performing normalization processing on the state sensitivity value, mapping it to a unit value interval, and outputting a dynamic aggregation weight factor.

[0012] Optionally, the online control parameter update module includes: an entropy calculation subunit, used to extract probability values ​​of the policy distribution from the real-time state action sequence, perform logarithmic operation on the probability values ​​and multiply them by the original probability values, and perform summation and inversion to generate an information entropy sequence; a statistical boundary construction subunit, used to construct a data queue based on the first-in-first-out rule as a sliding window, calculate the arithmetic mean and standard deviation of the values ​​in the data queue, and perform addition operation on the arithmetic mean and the standard deviation to obtain the uncertainty boundary; and a parameter trigger update subunit, used to perform numerical comparison between the current value of the information entropy sequence and the uncertainty boundary, trigger gradient vector superposition operation for the migration control parameters, and output optimized control parameters.

[0013] Optionally, the step of triggering the gradient vector superposition operation for the migration control parameters includes: when the current value is greater than the uncertainty boundary, performing chain differentiation using the policy distribution to calculate the gradient of the ascending direction in the current state and generating a parameter gradient vector; performing bitwise addition between the parameter gradient vector and the migration control parameters, and translating the coordinate position of the migration control parameters in the vector space to generate optimized control parameters.

[0014] Optionally, the consistency error calculation module includes: a weighted deviation operation subunit, used to calculate the Euclidean distance between the updated simulation state action sequence and the real-time state action sequence in the vector space, and perform a product operation between the Euclidean distance and the dynamic aggregation weight factor to generate a consistency error value; a dynamic threshold drift subunit, used to construct an error storage queue as a sliding window, and calculate the arithmetic mean of the error values ​​through the error storage queue to obtain a dynamic error threshold; and a sample loop closure correction subunit, used to perform a numerical comparison between the consistency error value and the dynamic error threshold, and when the consistency error value is greater than the dynamic error threshold, trigger a shearing iteration operation for the benchmark control parameter, and append the multidimensional feature vector to the historical multidimensional feature vector set.

[0015] Based on the same inventive concept, this invention also provides a cold chain sorting optimization method based on digital twins and reinforcement learning. The method includes: acquiring point cloud data and ambient temperature data of the target object; extracting geometric and material property features through model encoding; constructing a multidimensional feature vector; mapping the multidimensional feature vector to a vector space to generate a feature manifold state variable; initializing a digital twin simulation environment; performing action sampling based on the feature manifold state variable to obtain a simulation state action sequence; calculating the probability ratio between the current policy distribution and the sampled policy distribution; obtaining baseline control parameters through shearing iteration; extracting a migration multidimensional feature vector and a migration state action sequence from the target object; calculating the cosine similarity between the migration multidimensional feature vector and a preset historical multidimensional feature vector set one by one; and selecting the baseline control parameter corresponding to the maximum value as the initialization parameter. The loss function gradient is calculated based on the migration state action sequence, and gradient updates are performed on the initialization parameters to obtain migration control parameters. During the actual execution, real-time state action sequences are collected, and the information entropy sequence of the policy distribution is calculated. The mean and standard deviation of the information entropy sequence are calculated within a sliding window to construct an uncertainty boundary. When the information entropy sequence exceeds the uncertainty boundary, gradient updates are performed on the migration control parameters to generate optimized control parameters. An updated simulation state action sequence is generated using the optimized control parameters, and a consistency error is calculated between the optimized and real-time state action sequences. A dynamic error threshold is calculated using a sliding window. When the consistency error exceeds the dynamic error threshold, a shearing iteration is performed to correct the baseline control parameters. The multidimensional feature vector is recalculated and written into the historical multidimensional feature vector set.

[0016] Compared with the prior art, the present invention has the following advantages: 1. This solution addresses the issue of nonlinear drift in physical properties caused by temperature changes in cold chain environments, improving grasping stability. By employing point cloud analysis and temperature alignment units, it goes beyond relying solely on visual geometric features, incorporating joint encoding of environmental temperature time-series data and material properties. This enables the perception of the impact of low temperatures on the physical properties of packaging materials, overcoming the grasping slippage or damage issues caused by the "vision-touch" mismatch in cold chain scenarios, and enhancing the robot's robustness in variable temperature conditions.

[0017] 2. It achieves rapid cold start for new packaging objects with "zero samples" or "few samples," significantly reducing production line changeover costs. Utilizing the transfer loss gradient calculation module, it avoids retraining with massive amounts of data. Instead, it calculates the cosine similarity between the transfer features and the historical feature set, directly calling the most similar historical baseline parameters as initialization parameters. This similarity-based parameter initialization mechanism enables the robot to achieve millisecond-level policy transfer when facing frequently changing new packaged goods in cold chain warehousing, solving the pain points of "catastrophic forgetting" and slow convergence speed of traditional deep reinforcement learning models in multi-variety sorting scenarios.

[0018] 3. Ensured safe online control and stable system operation under uncertain environments. A baseline parameter update mechanism based on probability ratio shearing was constructed, and sub-units were built by combining entropy statistical boundaries. By limiting the magnitude and step size of policy updates, control policy oscillations caused by excessive exploration noise or erroneous samples were effectively prevented.

[0019] 4. A closed-loop feedback mechanism for "virtual-real consistency" was established, enabling continuous autonomous evolution throughout the entire lifecycle. A consistency error calculation module performs bidirectional verification between simulation predictions and real-time actions, and a dynamic threshold drift subunit automatically filters out normal environmental noise. Only when the error exceeds the dynamic threshold is a shearing iteration triggered and the historical feature library updated. This mechanism avoids redundant calculations of invalid data, achieves high-precision model calibration with low computational power consumption, and enables the model to continuously accumulate "experience" and automatically adapt to the long-term environmental evolution of cold chain warehouses as operating time increases.

[0020] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a framework diagram of the cold chain sorting optimization system and method based on digital twins and reinforcement learning, according to an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of the cold chain sorting optimization system and method based on digital twin and reinforcement learning according to an embodiment of the present invention.

[0024] Figure 3 This is a ternary graph showing the multi-objective weight distribution of different packaging objects in an embodiment of the present invention.

[0025] Figure 4 This is a Taylor diagram of the contact force evaluation before and after migration correction in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Reference Figure 1 One embodiment of the present invention proposes a cold chain sorting optimization system and method based on digital twins and reinforcement learning. By adopting a virtual-real migration and dynamic entropy weight feedback mechanism, it can realize online adaptive correction of control strategy and improve the sorting accuracy and robustness of robots in cold chain environment.

[0028] like Figure 2 As shown, the system in this embodiment specifically includes: The point cloud temperature feature encoding module is used to acquire point cloud data and ambient temperature data of the target object, extract geometric features and material property features through model encoding, construct multi-dimensional feature vectors, and map the multi-dimensional feature vectors to the vector space to generate feature manifold state variables. The digital twin strategy iteration module is used to initialize the digital twin simulation environment, perform action sampling based on the characteristic manifold state variables to obtain the simulation state action sequence, calculate the probability ratio between the current strategy distribution and the sampled strategy distribution, and obtain the baseline control parameters through shearing iteration. The migration loss gradient calculation module is used to extract the migration multidimensional feature vector and migration state action sequence of the target object, calculate the cosine similarity between the migration multidimensional feature vector and the preset historical multidimensional feature vector set one by one, select the benchmark control parameter corresponding to the maximum value as the initialization parameter, calculate the gradient of the loss function based on the migration state action sequence, and perform gradient update on the initialization parameter to obtain the migration control parameter. The online control parameter update module is used to collect real-time state action sequences during physical execution, calculate the information entropy sequence of the strategy distribution, calculate the mean and standard deviation of the information entropy sequence within a sliding window to construct an uncertainty boundary, and perform gradient update on the migration control parameters when the information entropy sequence breaks through the uncertainty boundary to generate optimized control parameters. The consistency error calculation module is used to generate an updated simulation state action sequence through the optimized control parameters, calculate the consistency error with the real-time state action sequence, calculate the dynamic error threshold through a sliding window, and when the consistency error exceeds the dynamic error threshold, perform a shearing iteration to correct the baseline control parameters, recalculate the multidimensional feature vector and write it into the historical multidimensional feature vector set.

[0029] Optionally, the point cloud temperature feature encoding module includes: The point cloud parsing and temperature alignment unit is used to parse the three-dimensional spatial coordinates and normal vector data in the point cloud data, and simultaneously extract the environmental temperature time series data aligned with the timestamp of the point cloud data; The system receives raw point cloud stream data from a depth camera or LiDAR and transforms it into a structured point cloud feature matrix using point cloud processing algorithms. This matrix contains the geometric information of the sampled points on the target object's surface, including the spatial coordinates of each point and the corresponding surface normal direction vector. The system reads the timestamp of the point cloud data acquisition and uses this timestamp as an index to retrieve corresponding environmental parameters, particularly temperature values, from the environmental monitoring database, thus establishing a time-series mapping between visual data and environmental data. This environmental monitoring database is a time-series database built based on real-time data uploaded by temperature and humidity sensors deployed in a cold chain environment.

[0030] For example, point cloud data containing 4096 sampling points is collected and constructed into a matrix of 4096 rows and 6 columns. The first 3 columns represent the X, Y, and Z coordinates of each point in the Cartesian coordinate system, in millimeters; the last 3 columns represent the normal vector components corresponding to that point. The acquisition timestamp of this frame of data is recorded as time T, and the cold storage environment data at time T is retrieved, for example, the current temperature is -18 degrees Celsius and the humidity is 45%.

[0031] The geometric and material feature encoding unit is used to perform volume integration and surface area calculation on the three-dimensional spatial coordinates to generate geometric features, and to obtain the elastic modulus parameter and friction coefficient parameter of the target object to generate material property features; Based on the spatial coordinate data obtained from the analysis, discrete volume integration is performed to estimate the volume of the target object, and a triangulation algorithm is applied to calculate the surface area. Simultaneously, the bounding box dimensions (length, width, and height) of the target object are extracted, and the volume, surface area, and dimensional data are combined to form a geometric feature set. For material properties, instead of relying on real-time visual estimation, a physical property database built based on experimental data from materials mechanics is accessed. Based on the identified object category and the current ambient temperature, the corresponding elastic modulus and friction coefficient parameters are retrieved from the database. This physical property database is a lookup table established through tensile and friction tests on various packaging materials under different temperature gradients.

[0032] The vector space mapping unit is used to perform channel concatenation on the geometric features, the material property features, and the ambient temperature time series data to construct a multidimensional feature vector, input the multidimensional feature vector into a preset encoder network, perform nonlinear feature mapping, and output the feature manifold state quantity.

[0033] A feature channel concatenation operation is performed to merge geometric feature data, queried material physical property data, and environmental temperature time-series data into a high-dimensional comprehensive feature vector. This comprehensive feature vector is then input into an encoder network. This encoder network is a graph neural network model trained based on historical captured samples. Through weight matrix multiplication between network layers and nonlinear activation function operations, data features are extracted, and a low-dimensional vector is finally output as the feature manifold state variable. For example, the ReLU function can be used as the nonlinear activation function.

[0034] For example, geometric features, such as dimensions of 30 cm long, 20 cm wide, and 20 cm high, are combined with material properties, such as the elastic modulus of cardboard at low temperatures, and environmental features, such as -18 degrees Celsius, to form a raw feature vector of length 64. This raw feature vector is then input into a network encoder containing three hidden layers. After processing, a feature manifold vector of length 128 is output.

[0035] Optionally, the digital twin strategy iteration module includes: The simulation sampling execution unit is used to load the target object in the digital twin simulation environment, calculate the probability distribution of the grasping action according to the characteristic manifold state quantity, and perform random selection according to the probability distribution to generate a simulation state action sequence. The digital twin engine loads the 3D mesh model and physical properties of the target object, receives the characteristic manifold state variables, and inputs them into a parameterized motion mapping function. This mapping function consists of multiple linear transformations and nonlinear activation functions, and its output corresponds to the statistical parameters of the motion space. For the continuous motion space of the robotic arm, the mean vector and covariance matrix of the Gaussian distribution are calculated; for discrete motion spaces such as suction cup start-stop, the probability vector of the classification distribution is calculated. Based on these statistical parameters, Monte Carlo sampling is performed to determine the current specific action, and the virtual robot is driven to perform this action in a simulation environment that includes rigid body dynamics and contact mechanics solvers. The state transition data during the execution process is recorded to form a sequence of data including the current state, the executed action, the immediate reward, and the next state.

[0036] For example, the input is a feature manifold vector with dimension 128. Six mean parameters and six standard deviation parameters controlling the grasping pose are calculated using a mapping function. A set of grasping instructions is sampled from the constructed six-dimensional joint Gaussian distribution. The grasping process is simulated in a simulation environment, and the trajectory and force conditions of the robot's end effector are recorded with a time step of 10 milliseconds.

[0037] The value deviation calculation unit is used to calculate the weighted sum of the immediate reward value and the next state prediction value through the simulated state action sequence, and to calculate the difference between the weighted sum and the current state prediction value to generate a time-series difference error. First, the immediate reward value is calculated based on physical simulation feedback. This immediate reward value is calculated by a weighted sum of three components: grasping stability, placement space utilization, and operational efficiency. The grasping stability component is determined by the displacement and holding time of the grasped object; the placement space utilization component calculates the ratio of the placed object's volume to the container's volume; and the operational efficiency component is negatively correlated with operation time and energy consumption. Next, a parameterized value estimation function is used to map the state characteristics of the next time step into a scalar form of future reward prediction. The immediate reward value is added to the future reward prediction multiplied by a discount factor to obtain the target value. The algebraic difference between this target value and the current reward prediction based on the current state characteristics is calculated to obtain the time-series difference error.

[0038] For example, when calculating the immediate reward value, if an object is successfully grasped and its displacement exceeds 50 mm and remains stable for more than 2 seconds, a base reward of 10 points is awarded; if the operation takes 4 seconds, a time penalty of 0.4 points is deducted; if the space utilization rate after placement is 80%, a space reward of 4 points is added; the total immediate reward is 13.6 points. A discount factor of 0.99 is set. If the value estimation function predicts the value of the next state to be 20 points, then the target value is 13.6 + 0.99 × 20 = 33.4 points. If the predicted value of the current state is 30 points, then the generated temporal difference error is 33.4 - 30 = 3.4. Figure 3 As shown in the figure, the weight distribution ternary graph of the multi-objective optimization strategy for different packaging objects automatically seeks the best among the three dimensions of grasping stability, operation efficiency and energy saving. The dark and large data points indicate that under the conditions of high success rate and fast convergence speed, it tends to maximize the comprehensive benefits through dynamic trade-offs.

[0039] The shearing update unit is used to calculate the probability ratio of the same action in the current update process and the sampling process, perform a shearing constraint operation on the probability ratio, perform gradient ascent calculation with the temporal difference error, and output the baseline control parameters.

[0040] Read the old policy distribution parameters saved at the sampling time and the new policy distribution parameters in the current iteration step, calculate the quotient of the probability density values ​​of the selected action under the old and new distributions, and obtain the probability ratio. Set a numerical clipping range centered at the value 1. Determine if the probability ratio exceeds this range; if it does, forcibly reset its value to the corresponding boundary value. Construct an optimization objective function, which includes two minimum comparisons: one is the product of the probability ratio and the temporal difference error, and the other is the product of the clipped ratio and the temporal difference error. Calculate the partial derivative of the objective function with respect to the control parameters to obtain the parameter gradient vector. Use this gradient vector to update the weight parameters in the action mapping function and the value estimation function until the parameters converge, and output the final baseline control parameters.

[0041] For example, the probability ratio for a certain grasping action is calculated to be 1.3. A truncation parameter is set to 0.2, allowing a range of 0.8 to 1.2. Since 1.3 is greater than 1.2, the ratio used for gradient calculation is truncated to 1.2. If the temporal difference error is positive 3.4, the calculated corrected gradient direction will increase the probability of the action occurring, but the increase is limited by the truncation value, thus preventing excessively large policy update steps from causing algorithm oscillations.

[0042] Optionally, the output reference control parameters include: A numerical clipping range is constructed with the unit value 1 as the center. It is determined whether the probability ratio falls within the numerical clipping range. When the probability ratio exceeds the numerical clipping range, the probability ratio is mapped to the boundary value of the numerical clipping range to generate a truncation ratio sequence. A truncation hyperparameter is set to limit the magnitude of policy updates, preventing performance crashes caused by excessive differences between the old and new policies. This truncation hyperparameter is set to 0.1 or 0.2. A closed interval centered at the value 1 is constructed based on this hyperparameter, with a lower bound of 1 minus the truncation hyperparameter and an upper bound of 1 plus the truncation hyperparameter. The probability ratios corresponding to each action are iterated over, and numerical comparisons are performed. If a probability ratio is greater than the upper bound, it is forcibly assigned the upper bound value; if a probability ratio is less than the lower bound, it is forcibly assigned the lower bound value; if it is within the interval, its original value remains unchanged. Through this step-by-step processing, a truncation ratio sequence with the same dimension as the original probability ratio sequence is generated.

[0043] For example, if the truncation hyperparameter is set to 0.2, the constructed numerical clipping range will be from 0.8 to 1.2. If the probability ratio of the current action is 1.5, it is determined to be greater than 1.2, so the generated truncation ratio is 1.2. If the probability ratio of another action is 0.9, it is determined to be between 0.8 and 1.2, so the generated truncation ratio is still 0.9.

[0044] Calculate the first product of the probability ratio and the time series difference error, and the second product of the truncation ratio sequence and the time series difference error, respectively. Select the minimum value between the first product and the second product to construct the objective function, and perform gradient update on the initialization parameters to output the baseline control parameters.

[0045] Element-wise vector multiplication is performed. First, the untruncated original probability ratios are multiplied by their corresponding temporal difference errors to obtain the first product, which represents the policy improvement gain under unconstrained conditions. Simultaneously, the truncated ratio sequence is multiplied by the same temporal difference error to obtain the second product, which represents the conservative policy improvement gain under constrained conditions. When constructing the objective function, a minimum selection operation is performed on each pair of the first and second products. The smaller of the two values ​​is selected as the final gradient-driven term. The partial derivatives of this driving term with respect to the weight parameters in the policy mapping function are calculated to generate the parameter gradient. Stochastic gradient descent is used to update the initial parameters along the gradient direction according to the learning rate. This process is repeated until convergence is achieved, and the final output set of weights serves as the baseline control parameters.

[0046] For example, suppose the temporal difference error is positive 10, indicating that the action should be encouraged. For a probability ratio of 1.5, the first product is 1.5 multiplied by 10, which equals 15; for a cutoff ratio of 1.2, the second product is 1.2 multiplied by 10, which equals 12. The minimum of 15 and 12, i.e., 12, is chosen as the objective function value. This means that although the original gain is high, the algorithm forces a conservative gain value of 12 to calculate the gradient, thereby limiting the step size of parameter updates and preventing drastic policy changes. The gradient vector calculated using this objective function value is multiplied by the learning rate of 0.001 and then superimposed on the current network weight matrix.

[0047] Optionally, the migration loss gradient calculation module includes: The feature similarity optimization subunit is used to perform a dot product operation on each vector in the set of historical multidimensional feature vectors and divide it by the product of the vector magnitudes to obtain a similarity value sequence, extract the maximum similarity value from the similarity value sequence, and use the benchmark control parameter of the maximum similarity value as an initialization parameter. The system receives the transition multidimensional feature vector of the current target object, which is the feature manifold state output by the encoder network. Simultaneously, it accesses the historical multidimensional feature vector set, which stores the feature vectors of different packaged objects successfully processed in past tasks and their corresponding convergence control parameters. A traversal calculation is performed, performing cosine similarity calculations between the transition multidimensional feature vector and each historical vector in the historical set. This calculation involves dot product operations and multiplication of vector magnitudes. The dot product result is divided by the magnitude product to obtain a normalized similarity value sequence. This sequence is then numerically sorted, and the element with the largest value is identified. The control parameters of the historical object corresponding to this maximum value are extracted and used as the initialization parameters for the current task. This historical multidimensional feature vector set is a knowledge base accumulated over long-term operation through continuous additions by the consistency error calculation module. The similarity calculation is performed according to the following cosine similarity formula: , in, Represents the current new packaging target With historically known packaging objects The similarity value between them ranges from negative 1 to positive 1. The multidimensional feature vector representing the current input is output in real time by the encoder network. Representing the first in the set of historical multidimensional feature vectors Each vector is stored in the database. The Euclidean norm, or modulus, represents a vector. Physically, it measures the consistency of orientation between two eigenvectors in a multidimensional manifold space. The closer the value is to 1, the more similar their geometric and physical properties are, and therefore the higher the transferability of historical experience.

[0048] For example, the current input feature vector is A. The historical database contains vectors B and C. The dot product of A and B is 0.8, and the product of their magnitudes is 1, resulting in a similarity of 0.8. The dot product of A and C is 0.3, and the product of their magnitudes is 1, resulting in a similarity of 0.3. The maximum similarity value of 0.8 is extracted. The control parameter set corresponding to vector B, which contains the weight matrix of the neural network, is indexed and loaded into the current policy network as the initial state.

[0049] The action residual gradient calculation subunit is used to calculate the numerical deviation between the predicted action distribution and the actual action distribution using the transition state action sequence and the initialization parameters, and generate a parameter gradient vector based on the numerical deviation through backpropagation. The initialization parameters are loaded into the policy network model, which receives a sequence of transition states and actions, consisting of a small number of state-action pairs. The state vectors from the sequence are input into the policy network for forward propagation, outputting a predicted action distribution. This predicted action distribution is then numerically compared with the actual actions recorded in the sequence. A loss function is constructed to quantify the difference between the two, i.e., the numerical bias. Based on this numerical bias, the partial derivatives of the loss function with respect to the current network parameters are calculated using the backpropagation algorithm, thereby generating the parameter gradient vector.

[0050] For example, the transition state action sequence contains 5 sets of samples. The first set of states is input into the network, and the output is the predicted grasp position coordinates. The square of the Euclidean distance between the predicted coordinates and the actual successful grasp coordinates is calculated as the loss value. The loss values ​​of the 5 sets of samples are averaged. The derivative of the network weights is taken to obtain a gradient vector with the same dimension as the weight matrix. This vector indicates the direction in which the parameters should be adjusted to reduce the prediction error.

[0051] The parameter migration correction subunit is used to calculate the product of the parameter gradient vector and the maximum similarity value to obtain the correction amount, perform vector difference operation on the initialization parameters and the correction amount, and output migration control parameters.

[0052] The parameter update calculation involves performing a scalar multiplication operation between the parameter gradient vector and the maximum similarity value. The product is defined as a correction factor. A vector subtraction operation is then performed, subtracting this correction factor from the initial parameters to obtain the updated parameter set, i.e., the migration control parameters. The rapid adaptation calculation of the parameters is based on the following update formula: , in, The migration control parameters represent the output, i.e., the network weights after adapting to the new packaging. This represents the initialization parameters, which are the baseline parameters for the most similar task retrieved from the history database. This represents the maximum similarity value, and its significance lies in serving as a dynamic step size or learning rate. When the similarity is high, A larger value indicates that historical experience is very reliable, allowing the model to make confidence updates with larger steps based on historical parameters; conversely, a smaller value indicates a smaller step size. Representative of the new packaging task The parameter gradient vector is generated by the action residual gradient calculation subunit. The formula reflects the gradient-based fast adaptation mechanism, which enables the transfer from historical policy to new policy to be completed with only a small number of samples.

[0053] For example, one of the weights in the initialization parameters is set to 0.5. The corresponding gradient value is calculated to be 0.1. The maximum similarity value is 0.8. The correction is calculated as 0.1 multiplied by 0.8, which equals 0.08. A difference operation is performed, subtracting 0.08 from 0.5, which equals 0.42. The updated value of 0.42 is then used as the corresponding weight value in the migration control parameters.

[0054] Optionally, the method further includes: Extract the Euclidean norm of the characteristic manifold state variables, calculate the absolute value of the temporal difference error, and multiply the Euclidean norm and the absolute value to generate a state sensitivity value. The system reads the feature manifold state variable, a multidimensional numerical vector representing the robot's current observation state. It then uses a linear algebra library to calculate the L2 norm of this vector, which is the square root of the sum of the squares of its components. Simultaneously, it reads the temporal difference error generated during the digital twin policy iteration process. The system performs an absolute value operation on this error value to eliminate the influence of sign. Finally, it performs a scalar multiplication operation, multiplying the L2 norm of the feature vector by the absolute value of the temporal difference error. This product is defined as the state sensitivity value, used to quantify the urgency of the robot's learning needs in a specific state.

[0055] For example, the state variable of the characteristic manifold is a vector of length 128, and its Euclidean norm is calculated to be 2.5. The corresponding temporal difference error is -4.0. The absolute value of the temporal difference error is calculated to be 4.0. Performing a multiplication operation, 2.5 multiplied by 4.0 yields 10.0. This 10.0 is the state sensitivity value.

[0056] The state sensitivity values ​​are normalized and mapped to a unit value range, and a dynamic aggregated weight factor is output.

[0057] The state sensitivity values ​​are input into a nonlinear mapping function whose domain covers the set of real numbers and whose range is strictly limited to a unit interval between 0 and 1. Through the mapping transformation of this function, the sensitivity values, which may have a large range, are compressed into a unit interval. The output value is the dynamic aggregation weight factor. This factor is used to adjust the calculation weight of the consistency error, reflecting the focus on highly sensitive states.

[0058] Optionally, the online control parameter update module includes: The entropy operation subunit is used to extract the probability value of the policy distribution from the real-time state action sequence, perform logarithmic operation on the probability value and multiply it by the original probability value, and perform summation and inversion to generate an information entropy sequence. The policy distribution data for the current time step is extracted from the real-time action sequence. For discrete action space, the predicted probability values ​​of each action category are directly extracted; for continuous action space, the determinant of the covariance matrix of the Gaussian distribution is extracted as the basis for measuring the probability density. The Shannon entropy calculation process is executed, iterating through each probability value and calculating its product with its natural logarithm. All products are summed, and the negative of the sum is taken to obtain a non-negative scalar value, i.e., information entropy. This value quantifies the degree of uncertainty of the current policy when facing a specific state; a higher entropy value indicates greater hesitation in the policy's action selection for the current state, i.e., higher uncertainty. This calculation process is repeated to generate the information entropy sequence of the time series.

[0059] The statistical boundary construction subunit is used to construct a data queue based on the first-in-first-out rule as a sliding window, calculate the arithmetic mean and standard deviation of the values ​​in the data queue, and perform an addition operation on the arithmetic mean and the standard deviation to obtain the uncertainty boundary. A fixed-length buffer is allocated in memory to construct a first-in, first-out (FIFO) data queue. Historical entropy values ​​are sequentially stored in this queue. When new data arrives and the queue is full, the earliest-entered data is removed, maintaining the queue length and thus forming a sliding window. Statistical calculations are performed on all entropy values ​​in the queue, calculating their arithmetic mean to represent the historical average uncertainty level and their standard deviation to represent the volatility of uncertainty. An addition operation is performed, summing the arithmetic mean and the standard deviation to obtain a dynamically changing value as the uncertainty boundary. This boundary reflects the upper limit of normal uncertainty that can be tolerated in the current operational phase.

[0060] For example, the sliding window length is set to 50. The current queue stores the entropy values ​​of the last 50 time steps. The average of these 50 values ​​is calculated to be 0.5, and the standard deviation is 0.1. An addition operation is performed, 0.5 plus 0.1 equals 0.6. 0.6 is set as the uncertainty boundary at the current time step.

[0061] The parameter-triggered update subunit is used to perform a numerical comparison between the current value of the information entropy sequence and the uncertainty boundary, triggering a gradient vector superposition operation for the migration control parameters, and outputting optimized control parameters.

[0062] The system executes numerical comparison logic to determine whether the calculated information entropy value at the current moment is strictly greater than the calculated uncertainty boundary. If the comparison result is true, it indicates that the current policy is facing an unprecedented complex state or abnormal situation, and is in a state of high uncertainty. At this time, a trigger signal is generated to start the parameter update process. The policy gradient vector for the current state is calculated, and this gradient vector is superimposed on the weight matrix contained in the migration control parameters with a preset step size to complete an online parameter correction, and the corrected optimized control parameters are output. If the comparison result is false, the current parameters remain unchanged.

[0063] For example, the currently calculated information entropy is 0.7, and the uncertainty boundary is 0.6. Since 0.7 is greater than 0.6, it is determined that the boundary has been breached. The policy gradient is calculated and superimposed on the current neural network weights, enabling the network to quickly adapt to this high uncertainty state, and the updated weights are output as optimization control parameters.

[0064] Optionally, the step of triggering the gradient vector stacking operation for the migration control parameters includes: When the current value is greater than the uncertainty boundary, chain differentiation is performed using the strategy distribution to calculate the gradient of the ascending direction in the current state and generate a parameter gradient vector. The system performs conditional logic checks to confirm that the real-time calculated information entropy value is strictly greater than the dynamically constructed uncertainty boundary value. Once the boundary is confirmed to be exceeded, the current computational graph state of the policy network is locked. The probability output node of the currently executed action in the policy distribution is identified. The natural logarithm is performed on this probability value. Applying the chain rule from calculus, this process backpropagates the derivative of the logarithmic probability from the output layer to the preceding hidden layers, calculating the partial derivative of the logarithmic probability with respect to each weight parameter in the migration control parameters layer by layer. These partial derivatives are arranged according to the storage order of the parameters in memory and combined into a high-dimensional vector. This vector is the parameter gradient vector, whose geometric meaning indicates the direction in the parameter space where the probability of the current action increases the fastest, i.e., the ascending direction.

[0065] For example, the information entropy at the current moment is 0.8, and the uncertainty boundary is 0.6, so the judgment condition is met. For the currently executed grabbing action, its predicted probability at the output layer of the policy network is obtained as 0.3. The gradients of the 10,000 weight parameters in the network are calculated. The calculation result generates a vector containing 10,000 floating-point numbers, which is the parameter gradient vector.

[0066] Perform a bitwise addition operation between the gradient vector of the parameters and the migration control parameters, and translate the coordinate position of the migration control parameters in the vector space to generate optimized control parameters.

[0067] The vector update operation aligns the parameter gradient vector with the currently stored transfer control parameters in terms of dimensions. First, a scalar multiplication is performed on the parameter gradient vector, multiplied by the online learning rate coefficient, to adjust the magnitude of the gradient. Then, element-wise addition is performed, adding the scaled gradient values ​​one by one to the weights corresponding to the transfer control parameters. Geometrically, this mathematical operation is equivalent to spatially translating the high-dimensional coordinates representing the network parameters along the direction of the gradient vector. The set of weights formed by the new coordinates after the translation constitutes the optimization control parameters. The online learning rate coefficient is a pre-defined hyperparameter based on stability requirements, typically set to a small value to ensure smooth fine-tuning.

[0068] For example, a specific weight in the migration control parameters is 0.500. The corresponding gradient vector component is 0.100. The online learning rate coefficient is 0.01. A calculation is performed: 0.500 is added to the product of 0.01 and 0.100, resulting in a new value of 0.501. After performing this operation on all parameters, the new set of parameters constitutes the optimization control parameters.

[0069] Optionally, the consistency error calculation module includes: The weighted deviation calculation subunit is used to calculate the Euclidean distance between the updated simulation state action sequence and the real-time state action sequence in the vector space, and to perform a product operation between the Euclidean distance and the dynamic aggregation weight factor to generate a consistency error value. The vector space distance calculation is performed by receiving two time-series data of the same dimension. One is an updated simulation state-action sequence re-generated in a digital twin environment based on optimized control parameters; the other is a real-time state-action sequence collected by sensors when the physical robot performs actions in a real physical environment. These two sequences are treated as two points in a high-dimensional space, and the Euclidean distance, i.e., the L2 norm, between them is calculated. The dynamic aggregation weighting factor is read, and a scalar multiplication operation is performed, multiplying the Euclidean distance by the dynamic aggregation weighting factor. This product is defined as the consistency error value. This value quantifies the degree of consistency between the current control strategy and reality, and has been weighted and amplified by the sensitivity factor. The consistency error value is calculated according to the following model: , in, This represents the generated consistency error value, i.e., the anomaly detection result. This represents the real-time state action sequence, i.e., the actual observation vector, which includes the robot's actual position, velocity, and force data. This represents the sequence of actions to update the simulation state, i.e., the digital twin prediction vector. It represents the Euclidean norm, used to calculate the magnitude of the difference between two vectors, and its physical meaning is the absolute deviation between the real and virtual vectors. This is a dynamic aggregation weighting factor, with a value ranging from 0 to 1. The physical meaning of the formula is that the weighting factor modulates the original deviation. When in a highly sensitive state, i.e., when the weighting factor is large, even a small physical deviation will be amplified into a larger consistency error, thereby improving the ability to detect anomalies.

[0070] The dynamic threshold drift subunit is used to construct an error storage queue as a sliding window, and to calculate the arithmetic mean of the error values ​​through the error storage queue to obtain the dynamic error threshold. A fixed-length error storage queue is maintained in memory. Each time a new error value is generated, it is appended to the tail of the queue. If the queue is full, the oldest data at the head is removed, forming a sliding window mechanism. Statistical analysis is performed on all historical error values ​​currently stored in the queue. The arithmetic mean of these values ​​is calculated to characterize the average deviation level under recent operating conditions. The standard deviation of these values ​​is also calculated to characterize the fluctuation range of the deviation. A dynamic error threshold is set based on the combination of the arithmetic mean and standard deviation. This threshold is not a fixed constant but drifts in real time with changes in operating conditions to accommodate normal fluctuations under different operating conditions. The dynamic error threshold is preferably determined using the following statistical formula: , in, This represents the dynamic error threshold. This represents the arithmetic mean of the historical error values ​​within the sliding window. This represents the standard deviation of historical error values ​​within the sliding window. The formula's value is based on the statistical properties of the normal distribution, meaning that under normal operating conditions, approximately 95% of error values ​​should fall within the range of the mean plus twice the standard deviation. Therefore, setting the threshold to this upper bound can effectively distinguish normal noise from genuine abnormal deviations.

[0071] For example, the sliding window length is set to 100. The queue stores the consistency error values ​​obtained from the past 100 calculations. The arithmetic mean of these 100 values ​​is calculated to be 2.0. The standard deviation is calculated to be 0.1. The calculation is performed: 2.0 plus 2 multiplied by 0.1 equals 2.2. 2.2 is set as the dynamic error threshold for the current moment.

[0072] The sample closed-loop correction subunit is used to perform a numerical comparison between the consistency error value and the dynamic error threshold. When the consistency error value is greater than the dynamic error threshold, a shearing iteration operation is triggered for the benchmark control parameter, and the multidimensional feature vector is appended to the historical multidimensional feature vector set.

[0073] Perform a comparison logic operation to determine whether the currently calculated consistency error value is strictly greater than the dynamic error threshold. The determination logic is shown in the following formula: , If this inequality holds, it indicates that the current virtual-to-real deviation has exceeded the statistically normal fluctuation range, indicating a model mismatch or environmental mutation. At this point, a trigger signal is generated. First, the shearing update logic is invoked to perform another shearing iteration operation on the current baseline control parameters, forcibly correcting the policy network's response to the current state. Second, the multi-dimensional feature vector at the current moment is extracted and added as a new high-value sample to the historical multi-dimensional feature vector set. This writing operation expands the knowledge base, enabling rapid retrieval of this experience through similarity matching when encountering similar high-error scenarios in the future. Figure 4 The image shows a Taylor plot of the digital twin simulation of the contact force of the robot grasping. The solid hexagon represents the migration-corrected model, which, compared to the uncalibrated model represented by the hollow inverted triangle, approximates the real physical observation benchmark shown by the solid rhombus in terms of correlation coefficient and standard deviation, verifying the accurate reproduction of the contact force characteristics.

[0074] For example, the current consistency error is 3.118. The calculated dynamic error threshold is 2.2. The comparison shows that 3.118 is greater than 2.2, indicating an anomaly. An iterative optimization of the baseline control parameters is immediately triggered. Simultaneously, the 128-dimensional feature vector describing this abnormal condition is stored in the historical database for transfer learning.

[0075] Based on the same inventive concept, this invention also provides a cold chain sorting optimization method based on digital twins and reinforcement learning, the method comprising: The point cloud data and ambient temperature data of the target object are acquired, geometric features and material property features are extracted through model encoding, a multidimensional feature vector is constructed, and the multidimensional feature vector is mapped to the vector space to generate the feature manifold state variables. Initialize the digital twin simulation environment, perform action sampling based on the characteristic manifold state variables to obtain the simulation state action sequence, calculate the probability ratio between the current policy distribution and the sampled policy distribution, and obtain the baseline control parameters through shearing iteration; The migration multidimensional feature vector and migration state action sequence are extracted from the target object. The cosine similarity between the migration multidimensional feature vector and the preset historical multidimensional feature vector set is calculated one by one. The baseline control parameter corresponding to the maximum value is selected as the initialization parameter. The gradient of the loss function is calculated based on the migration state action sequence, and the gradient is updated on the initialization parameter to obtain the migration control parameter. During the actual execution process, real-time state action sequences are collected, and the information entropy sequence of the strategy distribution is calculated. Within a sliding window, the mean and standard deviation of the information entropy sequence are calculated to construct an uncertainty boundary. When the information entropy sequence breaks through the uncertainty boundary, gradient updates are performed on the migration control parameters to generate optimized control parameters. The simulation state action sequence is updated by generating the optimized control parameters, and the consistency error is calculated with the real-time state action sequence. The dynamic error threshold is calculated by a sliding window. When the consistency error exceeds the dynamic error threshold, a shearing iteration is performed to correct the baseline control parameters, the multidimensional feature vector is recalculated and written into the historical multidimensional feature vector set.

[0076] It should be noted that the formulas described above, through the principle of dimensional consistency and mathematical standardization methods (such as normalization, dimensionless parameter conversion, or unit system unification), can translate physical quantities with different properties into unitless standard values ​​or parameters that can be superimposed in the same dimension. This eliminates the interference of different dimensions on the computational logic, allowing the formulas to retain the original data distribution characteristics while possessing mathematical rationality and adaptability to objective laws. These are conventional technical methods and will not be elaborated further. The electrical connections between the various units described above do not necessarily represent direct or indirect connections; any indirect connection method is applicable to the embodiments of this invention as long as it achieves the purpose of this invention. The above descriptions are merely exemplary embodiments of this invention and should not be construed as limiting the scope of this invention.

[0077] All equivalent changes and modifications made in accordance with the teachings of this invention shall still fall within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or customary techniques in the art not described herein.

Claims

1. A cold chain sorting optimization system and method based on digital twin and reinforcement learning, characterized in that, The system comprises: A point cloud temperature feature coding module is configured to obtain point cloud data and environment temperature data of a target object, extract geometric features and material attribute features through model coding, construct a multi-dimensional feature vector, map the multi-dimensional feature vector to a vector space, and generate a feature manifold state quantity; A digital twin strategy iteration module is configured to initialize a digital twin simulation environment, perform action sampling based on the feature manifold state quantity to obtain a simulation state action sequence, calculate a probability ratio of a current policy distribution and a sampled policy distribution, and obtain a baseline control parameter through clipping iteration; A migration loss gradient calculation module is configured to extract a migration multi-dimensional feature vector and a migration state action sequence for the target object, calculate a cosine similarity between the migration multi-dimensional feature vector and a preset historical multi-dimensional feature vector set one by one, select a baseline control parameter corresponding to a maximum value as an initialization parameter, calculate a loss function gradient based on the migration state action sequence, and perform gradient update on the initialization parameter to obtain a migration control parameter; An online control parameter update module is configured to collect a real-time state action sequence during real object execution, calculate an information entropy sequence of a policy distribution, calculate a mean value and a standard deviation of the information entropy sequence within a sliding window to construct an uncertainty boundary, perform gradient update on the migration control parameter when the information entropy sequence breaks through the uncertainty boundary, and generate an optimized control parameter; A consistency error calculation module is configured to generate an updated simulation state action sequence through the optimized control parameter, calculate a consistency error with the real-time state action sequence, calculate a dynamic error threshold value through a sliding window, and perform clipping iteration to correct the baseline control parameter when the consistency error breaks through the dynamic error threshold value, recalculate the multi-dimensional feature vector, and write the multi-dimensional feature vector into the historical multi-dimensional feature vector set.

2. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 1, wherein, The point cloud temperature feature coding module comprises: A point cloud analysis and temperature alignment unit is configured to analyze three-dimensional space coordinates and normal vector data in the point cloud data, and synchronously intercept environment temperature time series data aligned with a timestamp of the point cloud data; A geometric and material feature coding unit is configured to perform volume integration and surface area calculation on the three-dimensional space coordinates to generate geometric features, and obtain elastic modulus parameters and friction coefficient parameters of the target object to generate material attribute features; A vector space mapping unit is configured to perform channel splicing on the geometric features, the material attribute features, and the environment temperature time series data to construct a multi-dimensional feature vector, input the multi-dimensional feature vector into a preset encoder network, perform nonlinear feature mapping, and output a feature manifold state quantity.

3. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 1, wherein, The digital twin strategy iteration module comprises: A simulation sampling execution unit is configured to load a target object in the digital twin simulation environment, calculate a probability distribution of a grasping action according to the feature manifold state quantity, and generate a simulation state action sequence according to random selection based on the probability distribution; A value bias calculation unit is configured to calculate a weighted sum of an immediate reward value and a next state prediction value through the simulation state action sequence, calculate a difference between the weighted sum and a current state prediction value, and generate a time series difference error. The shear updating unit is configured to calculate a probability ratio of the same action in the current updating process and in the sampling process, perform a gradient ascent calculation on the probability ratio and the timing difference error, and output a reference control parameter.

4. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 3, wherein, The output reference control parameter comprises: a numerical shear range is constructed with the unit value 1 as the center, it is judged whether the probability ratio falls within the numerical shear range, and when the probability ratio exceeds the numerical shear range, the probability ratio is mapped to the boundary value of the numerical shear range to generate a truncated ratio sequence; a first product of the probability ratio and the timing difference error and a second product of the truncated ratio sequence and the timing difference error are calculated respectively, the minimum value of the first product and the second product is selected to construct a target function, and a gradient update is performed on the initialization parameter to output a reference control parameter.

5. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 1, wherein, The migration loss gradient calculation module comprises: a feature similarity optimization subunit configured to perform a dot product operation on the migration multi-dimensional feature vector and each vector in the historical multi-dimensional feature vector set, divide the product of the vector lengths to obtain a similarity value sequence, extract a maximum similarity value from the similarity value sequence, and take the reference control parameter of the maximum similarity value as an initialization parameter; an action residual gradient calculation subunit configured to calculate a numerical deviation between a predicted action distribution and an actual action distribution by using the migration state action sequence and the initialization parameter, and generate a parameter gradient vector by back propagation based on the numerical deviation; a parameter migration correction subunit configured to calculate a product of the parameter gradient vector and the maximum similarity value to obtain a correction amount, perform a vector difference operation on the initialization parameter and the correction amount, and output a migration control parameter.

6. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 3, wherein, The method further comprises: extracting a Euclidean norm of the feature manifold state quantity, calculating an absolute value of the timing difference error, performing a product operation on the Euclidean norm and the absolute value, and generating a state sensitivity value; performing a normalization process on the state sensitivity value, mapping to a unit value interval, and outputting a dynamic aggregation weight factor.

7. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 1, wherein, The online control parameter updating module comprises: an entropy value operation subunit configured to extract a probability value of a policy distribution from the real-time state action sequence, perform a logarithmic operation on the probability value, multiply the original probability value, perform a summation and negation, and generate an information entropy sequence; a statistical boundary construction subunit configured to construct a data queue as a sliding window based on a first-in-first-out rule, calculate an arithmetic mean value and a standard deviation of the values in the data queue, perform an addition operation on the arithmetic mean value and the standard deviation, and obtain an uncertainty boundary; a parameter trigger updating subunit configured to perform a numerical comparison between a current value of the information entropy sequence and the uncertainty boundary, trigger a gradient vector superposition operation on the migration control parameter, and output an optimized control parameter.

8. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 7, wherein, The trigger of the gradient vector superposition operation on the migration control parameter comprises: When the current value is greater than the uncertainty boundary, a chain rule is performed using the policy distribution to calculate an ascending direction gradient at the current state, and a parameter gradient vector is generated; A bitwise addition operation is performed on the parameter gradient vector and the transfer control parameter to shift the coordinate position of the transfer control parameter in the vector space to generate an optimized control parameter.

9. The digital twin and reinforcement learning based cold chain sortation optimization system and method of claim 6, wherein, The consistency error calculation module comprises: A weighted bias operator unit is configured to calculate the Euclidean distance between the updated simulation state action sequence and the real-time state action sequence in the vector space, perform a product operation on the Euclidean distance and the dynamic aggregation weight factor to generate a consistency error value; A dynamic threshold drift subunit is configured to construct an error storage queue as a sliding window, calculate the arithmetic mean of error values through the error storage queue to obtain a dynamic error threshold; A sample closed-loop correction subunit is configured to perform a numerical comparison on the consistency error value and the dynamic error threshold, trigger a clipping iteration operation on the reference control parameter when the consistency error value is greater than the dynamic error threshold, and append the multi-dimensional feature vector to the historical multi-dimensional feature vector set.

10. A cold chain sorting robot self-adaptive optimization method based on digital twinning and reinforcement learning, applied to the cold chain sorting optimization system and method based on digital twinning and reinforcement learning according to any one of claims 1-9, characterized in that, The method comprises: Point cloud data and environmental temperature data of a target object are obtained, geometric features and material attribute features are extracted through model encoding, a multi-dimensional feature vector is constructed, and the multi-dimensional feature vector is mapped to a vector space to generate a feature manifold state quantity; A digital twin simulation environment is initialized, an action sequence is obtained based on the feature manifold state quantity, a probability ratio of a current policy distribution and a sampling policy distribution is calculated, and a reference control parameter is obtained through clipping iteration; A transfer multi-dimensional feature vector and a transfer state action sequence are extracted from the target object, cosine similarities are calculated between the transfer multi-dimensional feature vector and a preset historical multi-dimensional feature vector set one by one, a reference control parameter corresponding to a maximum value is selected as an initialization parameter, a loss function gradient is calculated based on the transfer state action sequence, and a gradient update is performed on the initialization parameter to obtain a transfer control parameter; A real-time state action sequence is collected during real object execution, an information entropy sequence of a policy distribution is calculated, a mean value and a standard deviation of the information entropy sequence are calculated within a sliding window to construct an uncertainty boundary, and a gradient update is performed on the transfer control parameter when the information entropy sequence breaks through the uncertainty boundary to generate an optimized control parameter; An updated simulation state action sequence is generated through the optimized control parameter, a consistency error is calculated with the real-time state action sequence, a dynamic error threshold is calculated through a sliding window, and clipping iteration is performed to correct the reference control parameter when the consistency error breaks through the dynamic error threshold, the multi-dimensional feature vector is recalculated and written into the historical multi-dimensional feature vector set.

Images