A spiral ship unloader operation work order processing method and system based on reinforcement learning

Abstract

The application discloses a spiral ship unloader operation work order processing method and system based on reinforcement learning, and the method comprises the following steps: receiving an external operation work order of a spiral ship unloader, and generating executable task instructions; collecting real-time environment information of the spiral ship unloader operation environment; based on an Actor-Critic architecture reinforcement learning model, generating an operation strategy according to the task instructions and the real-time environment information; according to the operation strategy, adopting a heuristic search algorithm, taking path optimization, obstacle avoidance and efficiency optimization as priorities, planning an operation path and an action sequence of the spiral ship unloader, and controlling the spiral ship unloader to perform actions according to the operation path and the action sequence. Through the combination of the reinforcement learning intelligent decision and the heuristic search path planning, multi-target collaborative optimization of the whole process of the unattended spiral ship unloader operation is realized, and the operation efficiency, safety and self-adaptive capacity are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of spiral ship unloader control, in particular to a spiral ship unloader operation work order processing method and system based on reinforcement learning. BACKGROUND

[0002] In the modern port logistics system, the efficiency and automation level of bulk cargo loading and unloading are directly related to the throughput capacity and operating cost of the entire port. As the key equipment for bulk cargo unloading operation, the traditional operation mode of spiral ship unloader highly depends on the experience of operators for manual control. This mode not only requires high technical proficiency of the operators, who need to keep focused at all times to cope with the complex cabin environment and irregular cargo accumulation form, but also leads to a bottleneck in the efficiency of manual operation that is difficult to break through. At the same time, the complex environment in the cabin and the poor visibility, combined with long-time high-intensity manual operation, also bring high safety risks, which easily lead to equipment collision or personnel safety accidents, and has become a pain point that needs to be solved in the industry.

[0003] With the continuous development of industrial automation and artificial intelligence technology, the trend of unmanned and intelligent port equipment is becoming increasingly evident. In order to improve the safety and efficiency of operation, some spiral ship unloader systems that attempt to introduce automated control have appeared in the market. However, these existing systems generally have certain limitations. First, the data sources are often relatively single, usually relying only on pre-set programs or limited sensor information, making it difficult to comprehensively and accurately perceive and respond to the dynamically changing real operation environment. Second, the core decision-making algorithms of these systems are mostly based on simple pre-programmed rules or traditional control logic, lacking self-learning and self-adaptive optimization capabilities. When faced with unprecedented cargo stacking, new obstacle layout or unexpected situations, the performance of such systems is often unsatisfactory, with serious lack of flexibility, and even may lead to interruption of operation.

[0004] Specifically, the operation planning module of the existing system has poor flexibility, usually adopting static or limited-computing-capability path planning methods, which are difficult to calculate the optimal solution that takes into account the shortest path, obstacle avoidance safety and energy efficiency in real time in complex unstructured environments. The decision-making process is often fragmented, failing to form an organic closed loop of environment perception, intelligent decision-making, motion planning and safety control, resulting in limited improvement in overall operation efficiency and failing to fully meet the stringent requirements of modern ports for continuous and efficient unmanned operation. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a spiral ship unloader operation work order processing method and system based on reinforcement learning, which combines reinforcement learning intelligent decision-making with heuristic search path planning, realizes multi-objective collaborative optimization of the entire operation process of unmanned spiral ship unloader, and significantly improves the operation efficiency, safety and self-adaptive ability.

[0006] To solve the above technical problems, the first aspect of the embodiment of the present application provides a spiral ship unloader operation order processing method based on reinforcement learning, comprising the following steps:

[0007] Receiving an external operation order of a spiral ship unloader, and generating executable task instructions;

[0008] Collecting real-time environmental information of the spiral ship unloader operation environment;

[0009] Generating an operation strategy based on an Actor-Critic architecture reinforcement learning model according to the task instructions and the real-time environmental information;

[0010] According to the operation strategy, a heuristic search algorithm is used to prioritize path optimization, obstacle avoidance, and efficiency optimization, and the operation path and action sequence of the spiral ship unloader are planned, and the spiral ship unloader is controlled to perform actions according to the operation path and action sequence.

[0011] Further, the Actor-Critic architecture reinforcement learning model generates an operation strategy according to the task instructions and the real-time environmental information, comprising:

[0012] Fusing the task instructions and the real-time environmental information to construct a state vector representing the current state of the system;

[0013] Inputting the state vector into the policy network of the reinforcement learning model, and obtaining a multi-dimensional action vector through forward propagation calculation of the policy network;

[0014] The operation strategy is obtained based on the multi-dimensional action vector.

[0015] Further, the task instructions and the real-time environmental information are fused to construct a state vector representing the current state of the system, comprising:

[0016] Structurally analyzing the external operation order, extracting key parameters such as operation target position, cargo type, and operation priority, and generating standardized task instruction data;

[0017] Synchronously collecting the real-time environmental information, which includes cargo pile shape contour, obstacle coordinate set, and environmental temperature and humidity data;

[0018] Spatiotemporally aligning the task instruction data and the real-time environmental information to make all data correspond to the system state at the same decision-making time;

[0019] Using a weighted splicing method based on multi-source information fusion, the task instruction data and the real-time environmental information are combined into a preliminary feature vector;

[0020] Dimension reduction and normalization processing are performed on the preliminary feature vector to obtain the state vector.

[0021] Further, the state vector is input into the policy network of the reinforcement learning model, and a multi-dimensional action vector is obtained through forward propagation calculation of the policy network, including:

[0022] The state vector is input into the input layer of the policy network, and a standardized state vector is generated through a standardization preprocessing process;

[0023] The standardized state vector is input into the hidden layer, and deep control features are generated through a feature extraction and conversion process;

[0024] The deep control features are input into the output layer, and an initial multi-dimensional action vector is generated through an action generation process, each dimension element of the initial multi-dimensional action vector corresponding to a horizontal movement speed instruction, a vertical movement speed instruction, a rotation angle instruction, and a screw mechanism start-stop control signal required for controlling the screw ship unloader;

[0025] The initial multi-dimensional action vector is input into the output constraint process to generate a constrained action vector, each dimension value of the constrained action vector being within the effective control interval of the actuator;

[0026] The constrained action vector is input into the actuator adaptation process to generate a final multi-dimensional action vector, which is fully matched with the physical limit and response characteristics of the servo motor and screw mechanism.

[0027] Further, according to the operation strategy, a heuristic search algorithm is adopted to prioritize path optimization, obstacle avoidance, and efficiency optimization to plan the operation path and action sequence of the screw ship unloader, including:

[0028] Based on the operation strategy, the starting operation point and the target operation point of the screw ship unloader are determined, and the static obstacle map and dynamic obstacle prediction information in the current environment are obtained to generate environmental obstacle information;

[0029] Based on the environmental obstacle information, a composite cost function is constructed, which prioritizes minimizing the total path length, hard constrains safety obstacle avoidance, and optimizes the lowest operation energy consumption;

[0030] A heuristic search algorithm is used to start expanding path nodes from the starting operation point, calculate the comprehensive cost of each path node through the composite cost function, select the node with the lowest comprehensive cost as the current optimal path point, and finally generate a preliminary path;

[0031] Smoothly optimize the preliminary path to eliminate unnecessary turning and speed mutation points in the path, and obtain a continuous and smooth operation path conforming to the kinematic constraints of the spiral ship unloader;

[0032] Discretize the continuous and smooth operation path into a plurality of path points, determine a motion control instruction for each path point, the motion control instruction including a moving speed, a turning angle, and a spiral mechanism operation state, and generate an action sequence completely matching the operation path;

[0033] Add timestamp information based on the action sequence, and finally output a complete operation path and action sequence containing timestamp information.

[0034] Further, the constructing a composite cost function based on the environmental obstacle information comprises:

[0035] Determine the density and distribution characteristics of obstacles in the current operation environment based on the environmental obstacle information, and generate environmental feature parameters;

[0036] Configure a dynamic weight factor based on the environmental feature parameters, and generate a composite cost function framework with adaptive weight allocation;

[0037] Integrate an energy consumption evaluation model into the composite cost function framework to generate a multi-objective optimization function containing energy consumption prediction dimensions;

[0038] Balance the priority relationship between the three optimization objectives of path length, safety constraints, and energy consumption prediction through the dynamic weight factor, and generate the composite cost function.

[0039] Further, the smoothing optimization processing of the preliminary path comprises:

[0040] Apply a Bezier curve algorithm to the preliminary path for smoothing processing to generate a preliminary smooth path;

[0041] Perform kinematic feasibility verification on the preliminary smooth path based on the kinematic constraints of the spiral ship unloader to generate a verified path;

[0042] Perform curvature continuity processing on the verified path to generate a curvature continuous path;

[0043] Adjust the moving speed according to the curvature variation characteristics of the curvature continuous path to generate a speed-optimized path;

[0044] Perform second-order continuous derivable verification on the speed-optimized path to finally generate the continuous and smooth operation path.

[0045] Further, the adding timestamp information based on the action sequence comprises:

[0046] An initial time sequence is generated by assigning an initial time stamp to each path point based on the dynamic model of the spiral ship unloader;

[0047] A time-space mapping relationship is established for the initial time sequence to generate a time-coordinated sequence;

[0048] A buffer time mechanism is introduced at a key action conversion point in the time-coordinated sequence to generate an action sequence with a buffer time;

[0049] The action sequence with a buffer time is subjected to time synchronization optimization to generate a time-synchronized action instruction sequence;

[0050] The time-synchronized action instruction sequence is subjected to final timing verification to output a complete job path and action sequence with a time synchronization mark.

[0051] Correspondingly, a second aspect of the embodiment of the present application provides a spiral ship unloader job order processing system based on reinforcement learning, which processes external job orders based on the above-mentioned spiral ship unloader job order processing method based on reinforcement learning, comprising:

[0052] An order receiving module is configured to receive external job orders of the spiral ship unloader and generate executable task instructions;

[0053] An information collecting module is configured to collect real-time environmental information of the spiral ship unloader working environment;

[0054] A strategy generating module is configured to generate a job strategy based on an Actor-Critic architecture reinforcement learning model according to the task instructions and the real-time environmental information;

[0055] An action planning module is configured to plan a job path and action sequence of the spiral ship unloader according to the job strategy by using a heuristic search algorithm with path optimization, obstacle avoidance and efficiency optimization as priorities, and control the spiral ship unloader to perform actions according to the job path and action sequence.

[0056] Correspondingly, a third aspect of the embodiment of the present application provides an electronic device, comprising at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to make the at least one processor execute the above-mentioned spiral ship unloader job order processing method based on reinforcement learning.

[0057] Correspondingly, a fourth aspect of the embodiment of the present application provides a computer readable storage medium having computer instructions stored thereon, and the instructions are executed by a processor to implement the above-mentioned spiral ship unloader job order processing method based on reinforcement learning.

[0058] The above technical solutions of the embodiments of the present application have the following beneficial technical effects:

[0059] 1. By introducing a reinforcement learning model based on an Actor-Critic architecture, the system can autonomously generate an optimal operation strategy based on real-time environmental information and task instructions; the model has continuous learning and optimization capabilities, can effectively deal with complex unstructured environments such as irregular cargo pile shapes in the ship cabin and variable obstacle positions, and overcomes the rigidity and lack of flexibility of traditional pre-programmed systems, significantly improving the intelligent decision-making level and environmental adaptability of the spiral ship unloader in unattended working conditions;

[0060] 2. By multi-source information fusion to build an accurate state representation, and using a processing flow that includes output constraints and actuator adaptation, it is ensured that the action instructions output by the reinforcement learning strategy network meet the optimization goal and can be safely and accurately executed by the physical device; at the same time, the combination of heuristic search and refined path planning (including smoothing optimization and time sequence distribution) generates a space and time sequence that fully considers device dynamics constraints, thereby greatly ensuring the optimality of the operation path, the accuracy of action execution, and the safety and reliability of the entire operation process;

[0061] 3. The multi-objective optimization of path length, obstacle avoidance safety, and operation energy consumption is integrated into a compound cost function, and a dynamic weight mechanism is used for adaptive balancing, so that the planned operation path and action sequence are not only efficient in space and time, but also economical in energy consumption; the whole process automation from task reception to accurate execution is realized, reducing the efficiency loss and energy waste caused by human operation uncertainty and suboptimal decision-making, thereby significantly improving the overall operation efficiency and reducing the comprehensive operation and maintenance cost. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a flow chart of the spiral ship unloader operation work order processing method based on reinforcement learning provided by the embodiments of the present application;

[0063] Figure 2 is a block diagram of the spiral ship unloader operation work order processing system module based on reinforcement learning provided by the embodiments of the present application.

[0064] REFERENCE NUMERALS:

[0065] 1, work order receiving module, 2, information acquisition module, 3, strategy generation module, 4, action planning module. DETAILED DESCRIPTION

[0066] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application with reference to the specific embodiments and the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present application. In addition, in the following description, the description of the known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0067] Please refer to Figure 1 The first aspect of the embodiment of the present application provides a spiral ship unloader operation work order processing method based on reinforcement learning, comprising the following steps:

[0068] Step S100, receiving an external operation work order of the spiral ship unloader, and generating executable task instructions.

[0069] The operation instructions from the external system (such as the port management platform) are received through a secure communication protocol, and the received work order data is parsed and verified, and the core operation parameters are extracted, including the target operation position, the type of goods to be unloaded, the operation priority and other key information, and then the same is converted into standardized, structured executable task instructions recognizable and processable by the system internally, to provide accurate and standardized input basis for the subsequent decision-making process.

[0070] Step S200, collecting real-time environment information of the spiral ship unloader operation environment.

[0071] The multi-source sensors deployed in the operation site are used to continuously collect environment data related to the operation, and the specific collected information includes the cargo stack contour obtained through the visual sensor, the accurate coordinate information of static and dynamic obstacles obtained through the ranging radar, and the temperature and humidity parameters obtained through the environment sensor. The obtained raw data is preprocessed through a series of processes such as filtering, denoising, time synchronization and coordinate unification, and finally a complete and consistent real-time environment information data set describing the current operation environment state is formed.

[0072] Step S300, generating an operation strategy based on an Actor-Critic architecture reinforcement learning model according to the task instructions and the real-time environment information.

[0073] Firstly, the aforementioned generated standardized task instructions are fused with the preprocessed real-time environmental information to construct a high-dimensional state vector that can comprehensively represent the current state of the system. Subsequently, the state vector is input into a policy network trained using a deep deterministic policy gradient algorithm. The network calculates the output of a multi-dimensional continuous action vector through forward propagation. Each dimension of the vector accurately corresponds to a basic instruction for controlling the screw ship unloader actuator, such as horizontal and vertical movement speed, rotation angle, and screw mechanism start-stop state. Through the evaluation and guidance of the decision by the value network, an optimal operation strategy is finally generated, which takes into account both immediate operation benefits and long-term operation safety and efficiency.

[0074] Step S400, according to the operation strategy, a heuristic search algorithm is used to plan the operation path and action sequence of the screw ship unloader, with path optimization, obstacle avoidance and efficiency optimization as priorities, and the screw ship unloader is controlled to act according to the operation path and action sequence.

[0075] According to the operation strategy, a heuristic search algorithm is used to plan the operation path and action sequence of the screw ship unloader, with path optimization, obstacle avoidance and efficiency optimization as priorities, and the screw ship unloader is controlled to act according to the operation path and action sequence. This process first determines the starting and target positions of the operation according to the operation strategy, and constructs an environmental map containing all obstacles by integrating real-time environmental information. Subsequently, a heuristic search algorithm is used to plan a preliminary collision-free path in the configured search space, with a composite cost function that integrates path length, safe obstacle avoidance distance, and estimated energy consumption as the evaluation standard. The path needs to be further smoothed and optimized to eliminate unnecessary turns and speed jumps, and to strictly check whether it meets the kinematic constraints of the device. Finally, the optimized path is discretized into a series of path points with timestamps, and accurate motion control parameters are calculated for each point, thereby generating a detailed action sequence that is time and space synchronized and can directly drive the execution mechanism to complete the unloading operation.

[0076] Through the above-mentioned sequentially connected and tightly coupled technical steps, a complete closed-loop processing flow from intelligent decision-making to precise execution is constructed. The technical effect is that the system realizes autonomous intelligent control of the screw ship unloader unmanned operation in a complex and dynamic port operation environment, significantly improving the automation level of the operation process, the intelligence level of decision-making and planning, the self-adaptation ability to unstructured environment, and the safety and economy of the overall operation.

[0077] Specifically, the reinforcement learning model based on Actor-Critic architecture in step S300 generates an operation strategy according to task instructions and real-time environmental information, including:

[0078] Step S310, the task instruction and real-time environment information are fused and processed to construct a state vector representing the current state of the system.

[0079] The task instruction and real-time environment information are fused and processed to construct a state vector representing the current state of the system. First, the structured task instruction data after parsing is standardized to ensure that its numerical range matches that of the real-time environment information data. At the same time, the real-time environment information collected by the sensor is preprocessed, such as denoising and coordinate unification, to form a standardized environment data set. Then, a time-space alignment mechanism is used to ensure that the task instruction requirements and environment perception data are completely synchronized in terms of timestamp and spatial reference system. On this basis, a multi-source information fusion algorithm based on a dynamic weight factor is used to weight and splice the task target position, cargo type, priority instruction, and environmental information such as cargo pile contour, obstacle coordinates, and environmental parameters, and to combine the features to generate a high-dimensional preliminary feature vector. Finally, the feature vector is dimensionally reduced and normalized to eliminate redundant information and unify the dimensions, and a state vector with fixed dimensions is obtained, which can unambiguously and comprehensively represent the overall operation state of the spiral ship unloader system at the current time.

[0080] Step S320, the state vector is input into the policy network of the reinforcement learning model, and a multi-dimensional action vector is obtained through forward propagation calculation of the policy network.

[0081] The state vector is input into the policy network of the reinforcement learning model, and a multi-dimensional action vector is obtained through forward propagation calculation of the policy network. The state vector constructed in the above step is input into the policy network using the deep deterministic policy gradient algorithm. The input layer of the network first performs standardization preprocessing on the state vector to eliminate dimensional differences. Then the data is nonlinearly transformed through multiple hidden layers, each hidden layer is composed of fully connected neurons and enhances its ability to extract and understand complex state feature relationships through activation functions. The processed features are finally transmitted to the output layer, which generates a multi-dimensional continuous action vector according to the continuous control requirements of the spiral ship unloader. Each dimension element of the vector accurately corresponds to a basic control instruction, including horizontal movement speed, vertical movement speed, rotation angle, and start-stop control signal of the spiral mechanism.

[0082] Step S330, the operation strategy is obtained based on the multi-dimensional action vector.

[0083] The job strategy is obtained based on the multi-dimensional action vector. This step performs subsequent processing on the initial multi-dimensional action vector output by the strategy network to form a job strategy that can directly guide planning. First, the output constraint processing is performed on the numerical values of each dimension of the action vector to ensure that all control instruction values are within the effective operation interval of each actuator, preventing instruction over-limiting. Then, the actuator adaptation processing is performed, which scales and fine-tunes the instruction values according to the specific physical response characteristics, delay characteristics and physical limits of the servo motor and screw mechanism, to ensure that the final action vector generated can be accurately and smoothly executed by the physical system. At this point, the final multi-dimensional action vector constitutes a complete, reliable and executable job strategy.

[0084] Through the above steps S310 to S330, the conversion from multi-source heterogeneous information to executable control strategy is completed, and the technical effect is that through a set of rigorous data fusion, network reasoning and output adaptation process, it is ensured that the reinforcement learning decision result can accurately reflect the job intention and environment state, while strictly meeting the physical constraints of the actuator, thereby providing an optimized and feasible intelligent decision basis for subsequent path planning and action execution, significantly improving the overall decision reliability, control accuracy and job safety of the system.

[0085] Further, in step S310, the task instruction and real-time environment information are fused and processed to construct a state vector representing the current state of the system, including:

[0086] In step S311, the external job order is structurally parsed to extract the job target position, cargo type and job priority key parameters, and generate standardized task instruction data.

[0087] The external job order is structurally parsed to extract the job target position, cargo type and job priority key parameters, and generate standardized task instruction data. This process first performs syntax parsing and semantic analysis on the received original order data, identifies and extracts the key job parameters contained therein, including the three-dimensional coordinate information of the target cabin and unloading point, the specific type attributes of bulk cargo, and the job priority level identifier. Then, the heterogeneous original parameters are processed for data cleaning and format standardization, such as converting the coordinate system to the unloading machine base coordinate system, mapping the cargo type to a predefined category code, and converting the priority to a numerical quantitative indicator, to finally generate a standardized task instruction data that is structured, numerically specified and directly callable by subsequent algorithms.

[0088] In step S312, real-time environment information is synchronously collected, including cargo pile shape contour, obstacle coordinate set and environment temperature and humidity data.

[0089] Synchronization acquisition of real-time environmental information, including cargo pile shape contour, obstacle coordinate set and environmental temperature and humidity data. This process achieves the synchronization acquisition of multi-dimensional environmental data through the coordination of various sensing devices deployed in the work site. The visual sensor (such as a 3D camera or a laser radar) scans and obtains the three-dimensional point cloud data of the cargo surface in the cabin, and after processing, the accurate cargo pile shape contour geometric information can be obtained; the millimeter wave radar continuously detects the work area and outputs the coordinate set of static obstacles (such as cabin walls, supports) and dynamic obstacles (such as personnel, other equipment) relative to the ship unloader; the high-precision digital temperature and humidity sensor monitors the environmental temperature and humidity parameters in real time. All these data are given a unified time stamp, laying the foundation for subsequent fusion processing.

[0090] Step S313, spatio-temporal alignment of task instruction data and real-time environmental information, so that all data correspond to the system state at the same decision time.

[0091] Spatio-temporal alignment of task instruction data and real-time environmental information, so that all data correspond to the system state at the same decision time. This process is a key preprocessing step before data fusion. Spatio-temporal alignment contains two levels: time alignment ensures that all data samples used for decision making have the same time stamp, reflecting the system snapshot at the same time, which is usually achieved through data caching and synchronization triggering mechanism; spatial alignment unifies all data to a common coordinate system (usually with the screw ship unloader as the reference), so that the target position in the task instruction, the obstacle coordinates and the cargo contour in the environmental information have consistent spatial reference system, so that accurate relative relationship calculation can be performed.

[0092] Step S314, using a weighted splicing method based on multi-source information fusion, combining task instruction data and real-time environmental information into a preliminary feature vector.

[0093] Using a weighted splicing method based on multi-source information fusion, combining task instruction data and real-time environmental information into a preliminary feature vector. This process performs feature-level fusion of standardized task instruction data and preprocessed real-time environmental information. Fusion is not simply data stacking, but weighted splicing after assigning dynamic weight factors to different types of data, thereby combining into a high-dimensional preliminary feature vector. The weight factor is not fixed, but is adaptively adjusted according to the core target of the current work stage and the complexity of the environment, for example, increasing the weight of obstacle coordinate information in obstacle dense areas, and increasing the weight of target position information when approaching the target point, so as to highlight the most critical information features.

[0094] Step S315, dimension reduction and normalization processing of the preliminary feature vector to obtain a state vector.

[0095] The preliminary feature vectors are dimensionally reduced and normalized to obtain state vectors. This process optimizes the subsequent processing of the fused high-dimensional preliminary feature vectors to construct the final state vector. Dimension reduction processing (such as principal component analysis PCA or other feature selection methods) is used to eliminate redundant information and highly linearly correlated features in the feature vector, reduce the data dimension while preserving the significant features of the original data, thereby improving the computational efficiency of the subsequent reinforcement learning model and alleviating the dimension disaster problem. Subsequently, the dimensionally reduced feature data is normalized (such as Min-Max scaling or Z-Score standardization) to map the values of each dimension to a unified numerical interval (such as [0, 1] or [-1, 1]), to eliminate the problem of excessively large numerical differences caused by different physical dimensions, and to ensure that it is suitable as input for the neural network model. The final output is a state vector with moderate dimension and normalized values that can clearly and accurately represent the overall state of the system at the decision-making moment.

[0096] Through the above steps S311 to S315, a complete and efficient multi-source heterogeneous data fusion and state construction process is constructed. Through a series of rigorous data processing steps such as structured analysis, synchronous acquisition, spatio-temporal alignment, weighted fusion, and dimension reduction and normalization, the original work order information and environmental perception information with different meanings and dimensions are successfully converted into a high-quality state representation that can be effectively understood and processed by the reinforcement learning model, providing a solid data foundation for generating accurate and reliable work strategies, and fundamentally improving the perception accuracy and decision effectiveness of the intelligent decision system.

[0097] Further, the state vector is input into the policy network of the reinforcement learning model in step S320, and a multi-dimensional action vector is obtained through forward propagation calculation of the policy network, including:

[0098] In step S321, the state vector is input into the input layer of the policy network, and a standardized state vector is generated through the standardization preprocessing process.

[0099] The state vector is input into the input layer of the policy network, and a standardized state vector is generated through the standardization preprocessing process, which receives the state vector generated by the aforementioned data fusion step as the initial input of the policy network.

[0100] The input layer first performs standardization preprocessing on the state vector, usually using the Z-Score standardization method, which is based on the mean and standard deviation of each feature dimension obtained from historical data to center and scale the input data, converting it to a distribution with a mean of zero and a standard deviation of one. This eliminates the adverse effects of physical dimensions and numerical range differences on network training and inference, improves the stability and convergence speed of network training, and ultimately generates a standardized state vector with a numerical distribution that is more suitable for neural network processing.

[0101] Step S322, the normalized state vector is input into the hidden layer to generate deep control features through the feature extraction and conversion process.

[0102] The normalized state vector is input into the hidden layer to generate deep control features through the feature extraction and conversion process. This process feeds the normalized state vector into the hidden layer part of the policy network. The hidden layer is usually composed of multiple fully connected layers, each layer performs linear transformation (through weight matrix and bias vector) on its input data and applies a nonlinear activation function (such as ReLU) for conversion. Through this layer-by-layer processing, the network can gradually extract and combine the complex feature patterns in the input state, capture the deep and nonlinear mapping relationship between the state variables and the optimal control actions, and thus convert the original normalized state vector into a series of highly abstract and semantically rich deep control features, which constitute the high-level representation of the final decision action.

[0103] Step S323, the deep control features are input into the output layer to generate an initial multi-dimensional action vector through the action generation process, and each dimension element of the initial multi-dimensional action vector corresponds to the horizontal movement speed instruction, vertical movement speed instruction, rotation angle instruction and screw mechanism start-stop control signal required to control the screw unloader.

[0104] The deep control features are input into the output layer to generate an initial multi-dimensional action vector through the action generation process, and each dimension element of the initial multi-dimensional action vector corresponds to the horizontal movement speed instruction, vertical movement speed instruction, rotation angle instruction and screw mechanism start-stop control signal required to control the screw unloader. This process passes the deep control features output by the hidden layer to the output layer of the policy network. The number of neurons in the output layer matches the dimension of the action space according to the requirements of the continuous control task of the screw unloader. This layer performs the final linear or nonlinear transformation on the input features and directly generates a multi-dimensional continuous action vector.

[0105] Each dimension element of this vector is predefined to correspond to a specific basic control instruction: for example, the first dimension output represents the instruction value of the horizontal movement speed, the second dimension represents the instruction value of the vertical movement speed, the third dimension represents the instruction value of the body rotation angle, and the fourth dimension represents the state control signal (usually mapped to discrete start-stop states in a continuous value domain) of the screw mechanism start-stop. In this way, an initial multi-dimensional action vector inferred by the policy network is generated.

[0106] Step S324, the initial multi-dimensional action vector is input into the output constraint process to generate a constrained action vector, and the dimension values of the constrained action vector are all within the effective control interval of the actuator.

[0107] The initial multi-dimensional action vector is input into the output constraint flow to generate a constrained action vector, and the values of each dimension of the constrained action vector are within the effective control interval of the actuator, thereby imposing a hard constraint on the initial action vector output by the strategy network.

[0108] For each dimension of the action vector, the upper and lower limits of the effective value are set according to the explicit performance specifications and operating limits of the corresponding physical actuator (such as a servo motor or a screw drive). For example, the horizontal movement speed instruction must be limited between the maximum speed in the positive and negative directions that the motor can provide. This flow uses a clipping function to force any value in the initial action vector that exceeds the preset limit of the corresponding dimension to be within the limit, ensuring that all instruction values are theoretically executable, thereby generating a safe constrained action vector.

[0109] In step S325, the constrained action vector is input into the actuator adaptation flow to generate a final multi-dimensional action vector that fully matches the physical limits and response characteristics of the servo motor and screw mechanism.

[0110] The constrained action vector is input into the actuator adaptation flow to generate a final multi-dimensional action vector that fully matches the physical limits and response characteristics of the servo motor and screw mechanism. This process further considers the dynamic response characteristics and nonlinear factors of the actuator based on the output constraints, and makes fine adjustments to the instructions. This flow adjusts and scales the constrained action vector based on a pre-established actuator response model (which may consider factors such as response delay, nonlinear dead zone, and minimum control resolution). The purpose is to generate a final control instruction that not only can be executed by the physical system, but also ensures a higher matching degree and predictability between the instruction and the actual response of the mechanism, thereby improving the accuracy and stability of control, and finally outputting a final multi-dimensional action vector that perfectly matches the real physical system.

[0111] The above steps complete the accurate conversion from abstract state perception to specific and executable control instructions. Through the forward reasoning of the strategy network combined with rigorous post-processing of the output, the decision actions generated by the reinforcement learning agent not only reflect the intelligence based on environmental state optimization, but also strictly meet all physical constraints and actual dynamic response characteristics of the actuators, thereby achieving the best balance between theoretical optimality and practical feasibility, greatly enhancing the reliability, safety, and control performance of the entire intelligent decision system in real physical environments.

[0112] Specifically, in step S400, according to the operation strategy, a heuristic search algorithm is used to plan the operation path and action sequence of the screw unloader, with path optimization, obstacle avoidance, and efficiency optimization as priorities, including:

[0113] Step S410, determine the starting work point and target work point of the spiral ship unloader based on the work strategy, obtain the static obstacle map and dynamic obstacle prediction information in the current environment, and generate environment obstacle information.

[0114] The starting work point and target work point of the spiral ship unloader are determined based on the work strategy, the static obstacle map and dynamic obstacle prediction information in the current environment are obtained, and then comprehensive environment obstacle information is generated. The determination of the starting and target work points needs to be based on the task instruction and the current state of the equipment. The static obstacle map is usually derived from a pre-constructed environment model or real-time sensor scanning data. The dynamic obstacle prediction information relies on multi-frame sensor data fusion and motion trajectory to identify and estimate the position change trend of obstacles in the future period of time, thereby providing accurate environment perception basis for path planning.

[0115] Step S420, construct a composite cost function based on the environment obstacle information. The composite cost function takes the minimization of the total path length as the primary goal, takes safe obstacle avoidance as a hard constraint, and takes the minimization of work energy consumption as an optimization goal.

[0116] Based on the obtained environment obstacle information, a composite cost function is constructed. This function takes the minimization of the total path length as the primary optimization goal, and takes safe obstacle avoidance as a hard constraint to ensure that the path does not pass through any obstacle region. On this basis, the minimization of work energy consumption is further included as an optimization goal. By introducing an energy consumption model related to the motion state of the device, the system running cost is reduced while the work efficiency is met, and the overall economy is improved.

[0117] Step S430, a heuristic search algorithm is used to start from the starting work point and gradually expand the path nodes. The comprehensive cost of each path node is calculated by the composite cost function, the node with the lowest comprehensive cost is selected as the current optimal path point, and a preliminary path is finally generated.

[0118] A heuristic search algorithm is used to start from the starting work point and gradually expand the path nodes. The comprehensive cost of each candidate node is evaluated by the constructed composite cost function, the node with the minimum comprehensive cost is selected as the current expansion direction, and the process is gradually pushed forward until the target work point is reached, and finally a preliminary path is generated. This process needs to consider multiple factors such as path length, safety and energy consumption, to ensure that the generated path has feasibility and superiority in complex work environment.

[0119] Step S440, the preliminary path is smoothed and optimized to eliminate unnecessary turning points and speed mutation points in the path, and a continuous and smooth work path that meets the kinematic constraints of the spiral ship unloader is obtained.

[0120] The preliminary path is smoothed and optimized by introducing spline curve interpolation or Bezier curve fitting, etc. to eliminate sharp turns and sudden speed changes in the path, so that the path is continuous and smooth and meets the kinematic constraints of the spiral ship unloader, including maximum turning radius, acceleration limit, etc., thereby improving the stability and control accuracy of the device in actual operation.

[0121] In step S450, the continuously smooth operation path is discretized into a plurality of path points, the motion control instructions of each path point are determined, the motion control instructions include moving speed, turning angle and spiral mechanism operating state, and the action sequence completely matching the operation path is generated.

[0122] The continuously smooth operation path after optimization is discretized into a plurality of path points according to certain time or distance interval, and the corresponding motion control instructions of each path point are generated, including moving speed, turning angle, and start-stop and speed control instructions of the spiral mechanism, so as to constitute an action sequence completely matching the path, and ensure that the device can accurately and continuously perform the unloading operation.

[0123] In step S460, the timestamp information is added based on the action sequence, and the complete operation path and action sequence containing the timestamp information are finally output.

[0124] Based on the motion ability of the device and the operation requirement, each control instruction in the action sequence is assigned with corresponding timestamp information to form a complete operation plan with time sequence relationship, and the path and action sequence with timestamp are finally output, which provides clear and schedulable time reference and action basis for subsequent execution control.

[0125] The above path and action planning process significantly improves the operation safety, running efficiency and economy of the spiral ship unloader in complex port environment by fusing multi-objective optimization, dynamic obstacle prediction and kinematic constraint processing, and enhances the adaptability and decision reliability of the system to dynamic environment.

[0126] Further, the construction of the composite cost function based on the environmental obstacle information in step S420 includes:

[0127] In step S421, the obstacle density and distribution characteristics in the current operation environment are determined based on the environmental obstacle information, and the environmental feature parameters are generated.

[0128] Based on the environmental obstacle information, the current operation space is quantitatively analyzed, the number of obstacles in unit area and its spatial distribution characteristics are calculated, and the environmental feature parameters such as obstacle density and distribution characteristics are extracted; this process usually involves grid-based environment map and probability occupancy evaluation, and the environmental complexity and the difficulty of passing are represented in numerical form.

[0129] Step S422, configure a dynamic weight factor based on the environmental feature parameters to generate a composite cost function framework with adaptive weight distribution.

[0130] According to the acquired environmental feature parameters, the weight factor is dynamically configured, for example, the weight of safe obstacle avoidance is increased in a dense obstacle area, and in an open area, more attention is paid to the path length or energy consumption index, thereby constructing a composite cost function framework with environmental adaptability; the framework can flexibly adjust the contribution degree of different optimization targets according to the real-time environmental state, and enhance the decision rationality of the system in a variable scene.

[0131] Step S423, integrate the energy consumption evaluation model into the composite cost function framework to generate a multi-objective optimization function containing the energy consumption prediction dimension.

[0132] The energy consumption evaluation model constructed based on the device dynamics model and historical operation data is integrated into the composite cost function framework, and the model can predict the energy consumption under different action sequences and path forms, so that the composite cost function introduces energy consumption prediction as another important optimization dimension while optimizing the path length and safety, to realize true multi-objective collaborative optimization.

[0133] Step S424, balance the priority relationship among the path length, safety constraint and energy consumption prediction three optimization targets through the dynamic weight factor to generate a composite cost function.

[0134] The dynamic weight factor is used to adjust the trade-off relationship among the shortest path length, safe obstacle avoidance constraint and the lowest energy consumption three targets in real time, and a composite cost function is constructed which comprehensively reflects the actual operation demand; the function performs multi-dimensional evaluation on the candidate nodes in each step of the path search process, thereby guiding the search algorithm to generate a high-quality path which is not only the shortest in the geometric level, but also takes into account safety and economy.

[0135] The adaptive composite cost function can dynamically coordinate multiple optimization targets in a complex and variable operation environment, significantly improve the rationality, economy and reliability of path planning, and enhance the adaptive ability of the spiral ship unloader to different working conditions, thereby providing core decision support for efficient, safe and low-consumption autonomous operation.

[0136] Further, the smoothing optimization processing of the preliminary path in step S440 includes:

[0137] Step S441, apply a Bezier curve algorithm to the preliminary path for smoothing processing to generate a preliminary smoothed path.

[0138] The Bezier curve algorithm is applied to smooth the preliminary path, and the specific process includes: firstly, identifying the key turning points in the path as control points of the Bezier curve, and then inserting appropriate number of intermediate control points between adjacent control points according to the complexity of the path and the required smoothness; the smooth curve path is generated by calculating the parametric equation of the Bezier curve, ensuring the first derivative of the path to be continuous, and eliminating the angle mutation and jagged fluctuations in the original path.

[0139] In step S442, the kinematic feasibility of the preliminary smooth path is verified based on the kinematic constraints of the spiral ship unloader, and a verified path is generated.

[0140] The preliminary smooth path is verified in detail based on the kinematic constraints of the spiral ship unloader: a kinematic model containing parameters such as minimum turning radius, maximum angular velocity, and maximum acceleration of the device is established; the curvature radius of each point on the path is calculated and compared with the minimum turning radius of the device; the path segment that does not meet the kinematic constraints is identified, and the method of curve refitting or inserting a transition segment is used for adjustment; whether the adjusted path meets all kinematic constraints is verified, and a verified path that ensures the device can be actually executed is generated.

[0141] In step S443, the curvature continuity processing is performed on the verified path to generate a curvature continuous path.

[0142] The curvature continuity processing is performed on the verified path: a curve type with continuous curvature change such as Clothoid curve or quintic polynomial spline is used; appropriate transition curve segments are inserted at the detected curvature mutation points to ensure that the curvature change of the entire path is continuous and smooth; whether the processed path meets the requirements of curvature continuity is verified by numerical calculation method, and a path with continuous curvature characteristics is generated.

[0143] In step S444, the moving speed is adjusted according to the curvature change characteristics of the curvature continuous path to generate a speed optimized path.

[0144] The speed is adjusted according to the curvature change characteristics of the curvature continuous path: a curvature-speed mapping model considering the dynamic characteristics of the device is established, which includes factors such as maximum centripetal acceleration limit and driving system performance constraint; the travel speed is reduced in the path segment with large curvature to ensure the stability and safety of the device; the travel speed is increased in the straight line or large radius curve segment with small curvature to optimize the operation efficiency; the smooth changing speed profile is generated by considering the physical limits of acceleration and jerk, ensuring the smoothness of the device motion.

[0145] In step S445, the second-order continuous derivative verification is performed on the speed optimized path, and finally a continuous and smooth operation path is generated.

[0146] Strict second-order continuous and derivable verification of speed-optimized path: numerical differentiation method is used to calculate the first derivative (speed) and second derivative (acceleration) of the path position function; points or sections with discontinuous derivatives are detected and identified; further optimization processing is performed on these sections, such as higher-order spline interpolation or adjustment of speed planning parameters; finally, a smooth and continuous path is generated at the position, speed and acceleration levels, ensuring that the device can smoothly and accurately track and execute.

[0147] The above path smoothing optimization process not only significantly improves the geometric quality and smoothness of the generated path, but also ensures that the path fully complies with the kinematics and dynamics characteristics of the spiral unloader, enabling the device to complete the task in a more stable and efficient manner, while effectively reducing the impact load and energy consumption of the mechanical system, improving the operation efficiency, control accuracy and long-term reliability of the entire operation system.

[0148] Further, the step S460 of adding timestamp information based on the action sequence includes:

[0149] In step S461, an initial timestamp is assigned to each path point based on the dynamics model of the spiral unloader, and an initial time sequence is generated.

[0150] An initial timestamp is assigned to each path point based on the dynamics model of the spiral unloader, and an initial time sequence is generated. This process first calculates the shortest time required to complete each path movement under the condition of satisfying the device power constraints, based on the physical dynamics characteristics of the spiral unloader, including key performance parameters such as maximum acceleration, maximum deceleration and maximum sustainable speed of each motion axis, and the spatial distance between adjacent path points in the planned path. Based on this, the earliest arrival time of each point in the path sequence is calculated theoretically from zero time, thereby generating an initial time sequence containing the timestamps of each path point, which ensures the theoretical executability of the movement instructions.

[0151] In step S462, a time-space mapping relationship is established for the initial time sequence, and a time-coordinated sequence is generated.

[0152] A time-space mapping relationship is established for the initial time sequence, and a time-coordinated sequence is generated. The core of this process is to establish a strict time-space mapping relationship to ensure that the position instructions of the unloader at any time are accurately matched with its actual motion capability. The adjustment needs to ensure that the unloader can move at the calculated speed within the specified time to cover the corresponding spatial distance, thereby avoiding the disconnection between instructions and execution in time, and generating an action sequence that is completely coordinated and synchronized in time and space.

[0153] Step S463, introduce buffer time mechanism at key action transition points in the time-coordinated sequence, generate action sequence with buffer time.

[0154] Introduce buffer time mechanism at key action transition points in the time-coordinated sequence, generate action sequence with buffer time. This process identifies key action transition points in the time-coordinated sequence, such as the moment when the moving direction changes, the moment when the screw mechanism starts or stops, and the moment when the speed changes significantly.

[0155] Before these key points, a short buffer time segment can also be inserted, which is used to allow the mechanical system to smoothly transition to the new motion state, absorb the vibration and impact that may be caused by system inertia or minor control errors, and thus significantly improve the smoothness of action execution and equipment life.

[0156] Step S464, time synchronization optimization for action sequence with buffer time, generate time-synchronized action instruction sequence.

[0157] Time synchronization optimization for action sequence with buffer time, generate time-synchronized action instruction sequence. This process takes a global perspective of the system, ensuring that all execution mechanisms that need to work together (such as the large car walking motor, the small car walking motor, and the screw rotating motor) have their action instructions completely synchronized in time. For example, ensure that the instruction for the screw to start rotating is precisely aligned in time with the instruction for the screw mechanism to move to the exact position above the goods, avoiding situations such as empty rotation at the unloading point or not starting after reaching the destination. By fine-tuning and aligning the trigger times of each independent action instruction, a time-synchronized action instruction sequence is generated, in which all sub-actions are highly coordinated.

[0158] Step S465, final timing verification for time-synchronized action instruction sequence, output complete job path and action sequence with time synchronization mark.

[0159] Final timing verification for time-synchronized action instruction sequence, output complete job path and action sequence with time synchronization mark. This process serves as the last quality checkpoint of the timing planning, and conducts comprehensive logical and physical verification on the generated time-synchronized action instruction sequence. The verification content includes checking whether there are time conflicts (such as two contradictory actions scheduled at the same time), whether the timing of all actions still strictly meets all the dynamics subsystem constraints of the motion axes (such as whether the acceleration is out of limit), and whether the total job time meets the expected requirements. Only after passing all the verifications, the final complete job path and action sequence with precise time synchronization mark will be output, which can be directly issued to the control system for execution.

[0160] The above steps complete the whole process of assigning a precise time dimension to the space path action sequence, and the technical effect is that, through a series of rigorous time sequence planning steps such as initial allocation based on a dynamics model, space-time mapping coordination, key point buffering, multi-mechanism synchronization and final verification, the generated control instruction sequence is not only space path optimal, but also precise, smooth, coordinated and absolutely feasible in the time dimension, so as to ensure that the spiral ship unloader can automatically run in the most efficient, most stable and most reliable way in actual operation, and maximize the operation efficiency, equipment safety and control accuracy.

[0161] Correspondingly, please refer to Figure 2 The second aspect of the embodiment of the present application provides a spiral ship unloader operation work order processing system based on reinforcement learning, which processes external operation work orders based on the above-mentioned spiral ship unloader operation work order processing method based on reinforcement learning, comprising:

[0162] A work order receiving module 1 is configured to receive external operation work orders of the spiral ship unloader and generate executable task instructions;

[0163] An information collecting module 2 is configured to collect real-time environmental information of the spiral ship unloader operation environment;

[0164] A strategy generating module 3 is configured to generate an operation strategy based on an Actor-Critic architecture reinforcement learning model according to the task instructions and the real-time environmental information;

[0165] An action planning module 4 is configured to plan an operation path and an action sequence of the spiral ship unloader according to the operation strategy by using a heuristic search algorithm, with path optimization, obstacle avoidance and efficiency optimization as priorities, and control the spiral ship unloader to perform actions according to the operation path and the action sequence.

[0166] Correspondingly, the third aspect of the embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to make the at least one processor execute the above-mentioned spiral ship unloader operation work order processing method based on reinforcement learning.

[0167] Correspondingly, the fourth aspect of the embodiment of the present application provides a computer readable storage medium having computer instructions stored thereon, and the instructions are executed by a processor to implement the above-mentioned spiral ship unloader operation work order processing method based on reinforcement learning.

[0168] The embodiment of the present application aims to protect a spiral ship unloader operation work order processing method and system based on reinforcement learning, which has the following effects:

[0169] 1. By introducing an Actor-Critic architecture-based reinforcement learning model, the system can autonomously generate the optimal operation strategy based on real-time environmental information and task instructions; this model has the ability of continuous learning and optimization, which can effectively deal with complex unstructured environments such as irregular cargo pile shape in the cabin and variable obstacle position, overcoming the rigidity and lack of flexibility of traditional pre-programmed systems, and significantly improving the intelligent decision-making level and environmental adaptability of the spiral ship unloader in unattended working conditions;

[0170] 2. By multi-source information fusion to build an accurate state representation, and using a processing flow that includes output constraints and actuator adaptation, it is ensured that the action instructions output by the reinforcement learning strategy network not only meet the optimization goal but also can be safely and accurately executed by physical devices; at the same time, the combination of heuristic search and refined path planning (including smoothing optimization and time sequence distribution) generates a path in space and time sequence that fully considers the device dynamics constraints, thereby greatly ensuring the optimality of the operation path, the accuracy of the action execution, and the safety and reliability of the entire operation process;

[0171] 3. The multi-objective optimization of path length, obstacle avoidance safety, and operation energy consumption is integrated into the compound cost function, and the dynamic weight mechanism is used for adaptive balance, so that the planned operation path and action sequence are not only efficient in space and time, but also economical in energy consumption; the whole process automation from task reception to accurate execution is realized, reducing the efficiency loss and energy waste caused by human operation uncertainty and suboptimal decision-making, thereby significantly improving the overall operation efficiency and reducing the comprehensive operation and maintenance cost.

[0172] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be embodied in the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) having computer usable program code embodied therein.

[0173] The application is described with reference to flowcharts and / or block diagrams that illustrate the methods, apparatus (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1apparatuses that implement the functions specified in the flowchart or flowcharts and / or blocks. Figure 1

[0174] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart or flowcharts and / or blocks. Figure 1 apparatuses that implement the functions specified in the flowchart or flowcharts and / or blocks. Figure 1

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart or flowcharts and / or blocks. Figure 1 apparatuses that implement the functions specified in the flowchart or flowcharts and / or blocks. Figure 1

[0176] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the technical solutions of the present application. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.​​​

Claims

1. A spiral unloader operation order processing method based on reinforcement learning, characterized in that, The method comprises the following steps: receiving an external operation order of the screw ship unloader, and generating executable task instructions; collecting real-time environment information of the screw ship unloader operation environment; generating an operation strategy based on an Actor-Critic architecture reinforcement learning model according to the task instructions and the real-time environment information; based on the operation strategy, using a heuristic search algorithm to plan the operation path and action sequence of the screw ship unloader, with path optimization, obstacle avoidance and efficiency optimization as priorities, and controlling the screw ship unloader to perform actions according to the operation path and action sequence; the operation strategy based on the operation strategy, using a heuristic search algorithm to plan the operation path and action sequence of the screw ship unloader, including: determining the starting operation point and target operation point of the screw ship unloader based on the operation strategy, and obtaining the static obstacle map and dynamic obstacle prediction information in the current environment to generate environment obstacle information; based on the environment obstacle information, constructing a composite cost function, which takes the minimization of the total path length as the primary goal, safety obstacle avoidance as a hard constraint, and the minimization of operation energy consumption as an optimization goal; using a heuristic search algorithm to gradually expand path nodes from the starting operation point, calculating the comprehensive cost of each path node through the composite cost function, selecting the node with the lowest comprehensive cost as the current optimal path point, and finally generating a preliminary path; performing smoothing optimization processing on the preliminary path to eliminate unnecessary turning and speed mutation points in the path, and obtaining a continuous and smooth operation path that meets the kinematic constraints of the screw ship unloader; discretizing the continuous and smooth operation path into a plurality of path points, determining the motion control instructions of each path point, which include moving speed, turning angle and screw mechanism operation state, and generating an action sequence that completely matches the operation path; adding timestamp information based on the action sequence, and finally outputting a complete operation path and action sequence containing timestamp information.

2. The reinforcement learning-based spiral unloader job order processing method according to claim 1, characterized by, the Actor-Critic architecture reinforcement learning model generates an operation strategy based on the task instructions and the real-time environment information, including: fuse the task instructions and the real-time environment information, and construct a state vector representing the current state of the system; input the state vector into the policy network of the reinforcement learning model, and obtain a multi-dimensional action vector through forward propagation calculation of the policy network; based on the multi-dimensional action vector, the operation strategy is obtained.

3. The reinforcement learning-based spiral unloader job order processing method according to claim 2, characterized by, fuse the task instructions and the real-time environment information, and construct a state vector representing the current state of the system, including: structurally analyze the external operation order, extract the key parameters of operation target position, cargo type and operation priority, and generate standardized task instruction data; synchronously collect the real-time environment information, which includes cargo pile contour, obstacle coordinate set and environment temperature and humidity data; spatially align the task instruction data and the real-time environment information to make all data correspond to the system state at the same decision-making time; The task instruction data and the real-time environment information are combined into a preliminary feature vector by using a weighted splicing method based on multi-source information fusion; The preliminary feature vector is processed by dimension reduction and normalization to obtain the state vector.

4. The reinforcement learning-based spiral unloader job order processing method according to claim 2, characterized by, The state vector is input into the policy network of the reinforcement learning model, and a multi-dimensional action vector is obtained through forward propagation calculation of the policy network, including: The state vector is input into the input layer of the policy network, and a standardized state vector is generated through a standardization preprocessing process; The standardized state vector is input into the hidden layer, and deep control features are generated through a feature extraction and conversion process; The deep control features are input into the output layer, and an initial multi-dimensional action vector is generated through an action generation process, each dimension element of the initial multi-dimensional action vector corresponding to a horizontal movement speed instruction, a vertical movement speed instruction, a rotation angle instruction, and a screw mechanism start-stop control signal required for controlling the screw ship unloader; The initial multi-dimensional action vector is input into the output constraint process to generate a constrained action vector, and each dimension value of the constrained action vector is within the effective control interval of the actuator; The constrained action vector is input into the actuator adaptation process to generate a final multi-dimensional action vector, which is fully matched with the physical limit and response characteristics of the servo motor and screw mechanism.

5. The reinforcement learning-based spiral unloader job order processing method of claim 1, wherein, The composite cost function is constructed based on the environmental obstacle information, including: Based on the environmental obstacle information, the density and distribution characteristics of obstacles in the current working environment are determined to generate environmental feature parameters; Based on the environmental feature parameters, a dynamic weight factor is configured to generate a composite cost function framework with adaptive weight allocation; An energy consumption evaluation model is integrated into the composite cost function framework to generate a multi-objective optimization function including energy consumption prediction dimensions; The priority relationship between path length, safety constraints, and energy consumption prediction is balanced by the dynamic weight factor to generate the composite cost function.

6. The reinforcement learning-based spiral unloader job order processing method of claim 1, wherein, The preliminary path is processed by smoothing optimization, including: A preliminary smoothing path is generated by applying a Bezier curve algorithm to the preliminary path for smoothing processing; The kinematic feasibility of the preliminary smoothing path is verified based on the kinematic constraints of the screw ship unloader to generate a verified path; The curvature continuous path is generated by performing curvature continuous processing on the verified path; The speed optimization path is generated by adjusting the movement speed according to the curvature variation characteristics of the curvature continuous path; The continuous and smooth working path is finally generated by performing second-order continuous and derivable verification on the speed optimization path.

7. The reinforcement learning-based spiral unloader job order processing method of claim 1, wherein, The action sequence is added with timestamp information, including: An initial time sequence is generated by assigning an initial timestamp to each path point based on the dynamics model of the screw ship unloader; The initial time sequence is processed by establishing a time-space mapping relationship to generate a time-coordinated sequence; A buffer time mechanism is introduced at the key action transition points in the time-coordinated sequence to generate an action sequence with buffer time; The action sequence with buffer time is optimized by time synchronization to generate a time-synchronized action instruction sequence; The time synchronization action instruction sequence is subjected to final timing verification, and a complete job path and action sequence with a time synchronization mark are output.

8. A reinforced learning based spiral ship unloader job order processing system, characterized by, The spiral ship unloader job order processing method based on reinforcement learning according to any one of claims 1-7 processes external job orders, comprising: a job order receiving module for receiving external job orders of the spiral ship unloader and generating executable task instructions; an information collecting module for collecting real-time environmental information of the spiral ship unloader operation environment; a strategy generating module for generating an operation strategy based on an Actor-Critic architecture reinforcement learning model according to the task instructions and the real-time environmental information; an action planning module for planning an operation path and action sequence of the spiral ship unloader according to the operation strategy by using a heuristic search algorithm with path optimization, obstacle avoidance, and efficiency optimization as priorities, and controlling the spiral ship unloader to perform actions according to the operation path and action sequence.

9. An electronic device, comprising: comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform the spiral ship unloader job order processing method based on reinforcement learning according to any one of claims 1-7.

Images