Strong heterogeneous brick material stacking method and device, electronic equipment and storage medium

Abstract

The invention provides a strong heterogeneous brick material stacking method and device, electronic equipment and a storage medium, and relates to the technical field of intelligent constructions.The strong heterogeneous brick material stacking method comprises the steps that current feasible blocks in a to-be-stacked brick material list are stacked in a stacking space of a target kiln loading car on the basis of current action, space occupation information of the stacking space is determined, and the current state is determined; the current state and the current action are input into a brick stacking model, and the action value of each feasible action output by the brick stacking model is obtained; determining the next action based on the action value of each feasible action, and stacking the next feasible block in the to-be-stacked brick list into the stacking space; the brick material stacking model is obtained by performing reinforcement learning on the dual deep Q network based on a sample brick material list, sample actions corresponding to all feasible blocks in the sample brick material list and rewards corresponding to the sample actions. According to the method and device provided by the invention, the brick stacking efficiency is improved, and the space utilization rate of the kiln car is improved.

Description

Technical Field

[0001] The invention relates to the technical field of intelligent buildings, and in particular to a method, device, electronic equipment and storage medium for stacking highly heterogeneous bricks. Background Art

[0002] The brick stacking in the factory is mainly done by workers who stack the bricks on the kiln carts according to different brick models and their experience in manual stacking. Other workers are responsible for recording the stacking conditions such as brick type, quantity, number of damaged bricks and kiln cart number. However, since the process relies heavily on human judgment, the space utilization rate of the kiln cart cannot be maximized as much as possible, and the factory's logistics costs are difficult to effectively reduce.

[0003] Related technologies solve the brick stacking problem through genetic algorithms and simulated annealing algorithms, but they all have the problems of low efficiency in large-scale solutions, low stacking utilization rate and limited application scope.

[0004] Therefore, how to improve the efficiency of brick stacking and increase the space utilization of kiln loading vehicles has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0005] The invention provides a method, device, electronic equipment and storage medium for stacking highly heterogeneous bricks, which are used to solve the technical problem of how to improve the efficiency of brick stacking and the space utilization rate of kiln loading vehicles.

[0006] The present invention provides a method for stacking highly heterogeneous bricks, comprising: Based on the current action, currently feasible blocks in the list of brick materials to be stacked are stacked into the stacking space of the target kiln loading vehicle, space occupancy information of the stacking space is determined, and the current state is determined based on the space occupancy information; Input the current state and the current action into a brick stacking model to obtain the action value of each possible action output by the brick stacking model; Determine the next action based on the action value of each feasible action, and stack the next feasible block in the list of brick materials to be stacked into the stacking space based on the next action; Among them, the brick stacking model is obtained after reinforcement learning of a dual deep Q network based on a sample brick list, sample actions corresponding to each feasible block in the sample brick list and rewards corresponding to the sample actions; the reward is determined based on the space occupancy information of the stacking space.

[0007] In some embodiments, the brick stacking model is trained based on the following steps: Determine a set of feasible actions corresponding to a current training state and a current sample feasible block in the sample brick material list based on the space occupancy information in the palletizing space; Inputting the current training state and each actionable action in the action set into a main network in the dual deep Q network to obtain an action value of each actionable action output by the main network; Determining a current training action in the current training state based on the action value of each possible action; Based on the current training action, the current sample feasible block is stacked in the stacking space, the space occupancy information in the stacking space is updated, and the next training state and the reward corresponding to the current training action are determined; Determine the experience data corresponding to the current training state based on the current training state, the next training state, the current training action, the set of feasible actions corresponding to the current sample feasible block, and the reward corresponding to the current training action; Training the main network based on empirical data corresponding to a plurality of training states, and updating model parameters of the target network in the dual deep Q network based on the model parameters of the main network; The brick stacking model is determined based on the model parameters of the target network obtained after the model training phase.

[0008] In some embodiments, the determining of the current training state and the set of feasible actions corresponding to the current sample feasible block in the sample brick list based on the space occupancy information in the palletizing space includes: Based on the space occupancy information in the stacking space, determining the feasible blocks that have been stacked in the stacking space; Based on the edges of the stacked feasible blocks, the remaining space in the stacking space is divided to determine a plurality of remaining subspaces; sorting the plurality of remaining subspaces based on the Manhattan distance between each remaining subspace and the palletizing space to determine a plurality of optimal remaining subspaces; The four corners of the plurality of optimal remaining subspaces are traversed based on the placement direction of the current sample feasible block to determine a feasible action set corresponding to the current sample feasible block.

[0009] In some embodiments, after determining the current training state and the feasible action set corresponding to the current sample feasible block in the sample brick material list based on the space occupancy information in the palletizing space, the method further includes: The current training action in the current training state is randomly selected from the feasible action set corresponding to the current sample feasible block based on a greedy strategy.

[0010] In some embodiments, the reward is determined based on the following steps: Based on the space occupancy information of the palletizing space, determine the current space occupancy rate, the volume occupancy rate at the current maximum occupancy height, and the average height of the currently occupied space; A reward corresponding to a current action is determined based on the current space occupancy rate, the volume occupancy rate at the current maximum occupied height, and the average height of the current occupied space.

[0011] In some embodiments, the feasible blocks in the list of bricks to be stacked are determined based on the following steps: Processing the list of bricks to be stacked to determine multiple brick types; Generate two regular cubic bricks corresponding to each brick type by taking the upper bottom surface and the lower bottom surface of each brick type as the bottom surface; Based on the two regular cubic bricks corresponding to each brick type, feasible blocks in the list of bricks to be stacked are determined.

[0012] In some embodiments, the main network and the target network have the same model structure; the main network includes a feature extraction module, a feature splicing module and a feature recognition module; The feature extraction module includes a convolution submodule and a fully connected submodule connected in sequence; the convolution submodule includes a plurality of convolution layers connected in sequence; the fully connected submodule includes a plurality of fully connected layers connected in sequence; the feature recognition module includes a plurality of fully connected layers connected in sequence; The input of the feature extraction module is the current state; the input of the feature splicing module is the output of the feature extraction module and the current action; the input of the feature recognition module is the output of the feature splicing module.

[0013] The present invention provides a strong heterogeneous brick stacking device, comprising: A state determination unit, for stacking the currently feasible blocks in the list of brick materials to be stacked into the stacking space of the target kiln loading vehicle based on the current action, determining the space occupancy information of the stacking space, and determining the current state based on the space occupancy information; An action evaluation unit, used for inputting the current state and the current action into a brick stacking model to obtain the action value of each possible action output by the brick stacking model; An action determination unit, used for determining a next action based on the action value of each feasible action, and stacking the next feasible block in the list of brick materials to be stacked into the stacking space based on the next action; Among them, the brick stacking model is obtained after reinforcement learning of a dual deep Q network based on a sample brick list, sample actions corresponding to each feasible block in the sample brick list and rewards corresponding to the sample actions; the reward is determined based on the space occupancy information of the stacking space.

[0014] The present invention provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for stacking strongly heterogeneous bricks when executing the computer program.

[0015] The present invention provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for stacking strongly heterogeneous bricks is implemented.

[0016] The strongly heterogeneous brick stacking method, device, electronic device and storage medium provided by the present invention stack the current feasible blocks in the list of bricks to be stacked into the stacking space of the target kiln loading vehicle according to the current action, determine the space occupancy information of the stacking space, and determine the current state according to the space occupancy information; input the current state and the current action into the brick stacking model to obtain the action value of each feasible action output by the brick stacking model; determine the next action according to the action value of each feasible action, and stack the next feasible block in the list of bricks to be stacked into the stacking space according to the next action; because the brick stacking model is obtained by reinforcement learning of the dual deep Q network based on the sample brick list, the sample actions corresponding to each feasible block in the sample brick list and the rewards corresponding to the sample actions, it can replace manual decision-making, learn the optimal strategy through interaction with the environment in a dynamic environment, determine the action, and can quickly and efficiently stack bricks, thereby improving the efficiency of brick stacking and improving the space utilization of the kiln loading vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is one of the flow diagrams of the method for stacking highly heterogeneous bricks provided by the present invention.

[0020] Figure 2 This is a data storage format diagram of the brick material list provided by the present invention.

[0021] Figure 3 It is an overall framework diagram of the highly heterogeneous brick stacking method provided by the present invention.

[0022] Figure 4This is the second schematic flow diagram of the method for stacking highly heterogeneous bricks provided by the present invention.

[0023] Figure 5 It is a model structure diagram of the dual deep Q network provided by the present invention.

[0024] Figure 6 It is a schematic structural diagram of the highly heterogeneous brick stacking device provided by the present invention.

[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units or modules is not necessarily limited to those steps or units or modules that are clearly listed, but may include other steps or units or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1 FIG. 1 is one of the flow diagrams of the method for stacking highly heterogeneous bricks provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0029] Step 110: stack the currently feasible blocks in the list of brick materials to be stacked into the stacking space of the target kiln loading vehicle based on the current action, determine the space occupancy information of the stacking space, and determine the current state based on the space occupancy information.

[0030] Specifically, the execution subject of the strong heterogeneous brick stacking method provided in the embodiment of the present invention is a strong heterogeneous brick stacking device. The device can be implemented by software, such as a brick stacking program; or by hardware, such as a computer or server that executes the strong heterogeneous brick stacking method.

[0031] The application scenario of the method provided in the embodiment of the present invention is to stack unordered bricks (bricks) of different models into a kiln loading vehicle. The target kiln loading vehicle is the kiln loading vehicle that currently needs to stack bricks. The stacking space is the space in the kiln loading vehicle that accommodates bricks. The kiln loading vehicle can be understood as a platform without any obstructions on all sides. The stacking space is a cube with the kiln loading vehicle as the bottom and the maximum kiln loading height as the height. When the kiln loading vehicle is fully loaded, the bricks on the top of the kiln loading vehicle are arched.

[0032] The list of bricks to be stacked is used to record information such as the brick type, size and quantity of the bricks. For example, the list of bricks to be stacked can store the information of the bricks in Excel format. The brick types can include rectangles, parallelograms, right-angled trapezoids and isosceles trapezoids. The size information can include length, width, height and difference (the coordinate difference between the upper and lower bottom surfaces of the brick in the horizontal direction).

[0033] The feasible blocks are regular cubic bricks generated according to the size information of the bricks, which is convenient for determining the stacking position, stacking direction and calculating the space occupation information of the stacking space in the method provided in the embodiment of the present invention. The current feasible block is the feasible block that needs to be stacked at present. The space occupation information is the space usage information of the stacking space occupied for stacking bricks.

[0034] Figure 2 This is a data storage format diagram of the brick material list provided by the present invention, such as Figure 2 As shown, by cleaning and extracting the data of the list of brick materials to be stacked, a feasible block set can be obtained. The feasible block set can be represented by n dictionaries. Each dictionary can include the brick type number, the upper bottom cube size information, the lower bottom cube size information and the brick type margin, etc.

[0035] The current action is an action execution command to stack the current feasible block into the stacking space of the target kiln loading vehicle. After stacking the current feasible block in the list of bricks to be stacked into the stacking space of the target kiln loading vehicle according to the current action, the space occupancy information of the stacking space can be calculated according to the volume of the current feasible block and the original space volume in the stacking space. The space occupancy information of the stacking space after the current feasible block is placed can be used as the current state.

[0036] The height map method is used to represent the occupied height of each point in the current palletizing space in millimeters to characterize the space occupancy situation. The current upper bottom surface height map and lower bottom surface height map are used to jointly represent the current environment status.

[0037] The update of the height map is set after each action is executed, that is, after the block is loaded into the space. When placing in layers, when the current layer is full and the next layer is placed, the height map of the lower bottom cube at this time is replaced by the current upper bottom cube height map, and the lower bottom cube height map remains unchanged.

[0038] Step 120, input the current state and the current action into the brick stacking model to obtain the action value of each feasible action output by the brick stacking model. The brick stacking model is obtained by reinforcement learning of the dual deep Q network based on the sample brick list, the sample actions corresponding to each feasible block in the sample brick list, and the rewards corresponding to the sample actions; the rewards are determined based on the space occupancy information of the stacking space; the dual deep Q network includes a main network and a target network; the target network is used to evaluate the main network.

[0039] Specifically, in reinforcement learning, the agent is used to stack multiple feasible blocks into the stacking space of the target kiln loading car. The state represents the environmental conditions or a description of the environment in which the agent is located at a certain moment. The state contains all the information of the environment, allowing the agent to make decisions based on the current state. Actions are operations or behaviors that the agent can perform. By selecting actions, the agent interacts with the environment, affects the state of the environment, and obtains corresponding feedback. Reward is the feedback of the environment to the agent's execution of an action, indicating the immediate return obtained by the agent after executing the action.

[0040] In an embodiment of the present invention, a strongly heterogeneous brick stacking device can be used as an intelligent agent. A double deep Q network (DDQN) is used as the initial model, and reinforcement learning training is performed on the double deep Q network according to a sample brick list, sample actions corresponding to each feasible block in the sample brick list, and rewards corresponding to the sample actions to obtain a brick stacking model. In the reinforcement learning process, the state can be the space occupancy information of the stacking space, the action can be the action execution command to stack the feasible blocks into the stacking space, and the reward can be determined according to the space occupancy information of the stacking space, such as space utilization, etc.

[0041] The dual deep Q network consists of a main network (model) and a target network (target model). In reinforcement learning, Q value (Action-Value Function) is used to evaluate the value of taking a specific action in a certain state. Q value can also be called state-action value, or action value, and is expressed as: represents the Q value, The following is the Bellman equation, which calculates the expected value; s represents the current state , a represents an action in the current state , is a variable, Represents the reward at the current moment, which is obtained by taking the current action at the current moment. Represents the reward at the next moment, and so on; Represents the discount factor, which is a constant between 0 and 1, and measures the impact of subsequent rewards on the current state value.

[0042] The main network is used to generate Q values ​​(which are action values) and select actions based on the current state. The target network is used to calculate the target Q value and reduce the overestimation bias of the Q value generated by the main network. By using two networks, DDQN can provide a more stable training process and reduce the overestimation problem in DQN. The introduction of the target network is to decouple the value assessment and action selection in the training process to avoid the occurrence of overestimation. The Q value generated by the main network is used to select actions, and the target network calculates the target Q value of the current state. These two Q values ​​are used together to calculate the loss function, which is used to reversely update the network parameters of the main network.

[0043] The current state and the current action are input into the brick stacking model as a set of state-action pairs to obtain the action value (that is, Q value) of each possible action output by the brick stacking model.

[0044] Step 130: determine the next action based on the action value of each feasible action, and stack the next feasible block in the list of brick materials to be stacked into the stacking space based on the next action.

[0045] Specifically, the brick stacking model can output multiple feasible actions, each of which has a corresponding action value. The feasible action with the highest action value can be selected as the next action. According to the next action, the next feasible block in the list of bricks to be stacked is stacked into the stacking space.

[0046] The strongly heterogeneous brick stacking method provided by the embodiment of the present invention stacks the current feasible blocks in the list of bricks to be stacked into the stacking space of the target kiln loading vehicle according to the current action, determines the space occupancy information of the stacking space, and determines the current state according to the space occupancy information; inputs the current state and the current action into the brick stacking model to obtain the action value of each feasible action output by the brick stacking model; determines the next action according to the action value of each feasible action, and stacks the next feasible block in the list of bricks to be stacked into the stacking space according to the next action; because the brick stacking model is obtained by reinforcement learning of the dual deep Q network based on the sample brick list, the sample actions corresponding to each feasible block in the sample brick list and the rewards corresponding to the sample actions, it can replace manual decision-making, learn the optimal strategy through interaction with the environment in a dynamic environment, determine the action, and can quickly and efficiently stack bricks, thereby improving the efficiency of brick stacking and improving the space utilization of the kiln loading vehicle.

[0047] It should be noted that each implementation of the present invention can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0048] In some embodiments, the brick stacking model is trained based on the following steps: Determine a set of feasible actions corresponding to a current training state and a current sample feasible block in a sample brick material list based on space occupancy information in the palletizing space; Input the current training state and each actionable action in the action set into the main network in the dual deep Q network, and obtain the action value of each actionable action output by the main network; Determine the current training action in the current training state based on the action value of each possible action; Based on the current training action, the current sample feasible block is stacked into the stacking space, the space occupancy information in the stacking space is updated, and the next training state and the reward corresponding to the current training action are determined; Determine the experience data corresponding to the current training state based on the current training state, the next training state, the current training action, the set of feasible actions corresponding to the current sample feasible block, and the reward corresponding to the current training action; The main network is trained based on the empirical data corresponding to the multiple training states, and the model parameters of the target network in the dual deep Q network are updated based on the model parameters of the main network; The brick stacking model is determined based on the target network obtained after the model training phase.

[0049] Specifically, the training process of the brick stacking model mainly includes generating empirical data and learning based on the empirical data.

[0050] The process of generating empirical data includes: According to the space occupancy information in the palletizing space, the set of feasible actions corresponding to the current training state and the current sample feasible block in the sample brick list is determined. For feasible blocks, the placement point and placement direction in the palletizing space will be different, so a set of feasible actions consisting of multiple feasible actions of a certain layer can be generated.

[0051] Input the current training state and each actionable action in the action set into the main network in the dual deep Q network to obtain the action value (Q value) of each actionable action output by the main network. According to the action value of each actionable action, determine the current training action in the current training state. For example, the actionable action with the highest action value can be determined as the current training action. According to the current training action, stack the current sample feasible block into the stacking space, update the space occupancy information in the stacking space, and determine the next training state and the reward corresponding to the current training action.

[0052] Determine the experience data corresponding to the current training state according to the current training state, the next training state, the current training action, the set of feasible actions corresponding to the current sample feasible block, and the reward corresponding to the current training action.

[0053] The process of learning from empirical data involves: The main network is trained according to the empirical data corresponding to the multiple training states, and the model parameters of the target network in the dual deep Q network are updated according to the model parameters of the main network, and the brick stacking model is determined according to the model parameters of the target network.

[0054] Figure 3 It is the overall framework diagram of the strong heterogeneous brick stacking method provided by the present invention, such as Figure 3 As shown, the strongly heterogeneous brick stacking method can include two parts: model training and search stacking.

[0055] The model training phase is described in detail below.

[0056] Step S1, obtain a sample brick material list. Read a sample brick material list stored in Excel format (for example, it may contain 50 sets of brick models to be stacked), and after sorting, a feasible block set represented by a dictionary can be obtained. Each dictionary represents a round of training data.

[0057] Step S2, create the main network (model) and target network (target model) in the dual deep Q network. Set the target network to evaluate the main network. The two have the same structure, built in the order of five convolutional layers, three fully connected layers (with a Dropout layer in the middle), one splicing layer, and two fully connected layers. A pooling layer is added after the first two convolutional layers, and ReLU is used as the activation function. The Adam optimizer is used as the optimizer of the model, and the learning rate is set to 0.01. The model input is state and action, and the output is Q value. The data set obtained in step S1 is used as input, and 50 cycles of step S3 are performed.

[0058] Step S3, initialize the experience pool, training step length and experience playback times. Each piece of experience data in the experience pool is stored in the form of "current training state, next training state, current training action, feasible action set corresponding to the current sample feasible block, reward corresponding to the current training action, and end flag". Clear the experience pool and set the data upper limit of the experience pool to 10,000. When the experience pool is full, set new experience data to overwrite the old experience data (starting from the first piece of experience data). Set the training step length to 0, and perform the following operations in each round (50 times in total).

[0059] Step S3.1, initialize the state (state), end flag (done), height map (u_plain and l_plain), and experience replay times in the environment. The state is a stack of two height maps (which can represent the space occupancy information of the palletizing space). The end flag is initially 0. If the palletizing space is full, it means that this round of training is over, and the end flag (done) is set to 1. The height map is represented by a matrix, and the number at each position indicates the height currently occupied by the brick at the current position. The height map is set to a full 0 matrix of 1610*1610 (i.e., the length * width of the palletizing space, in mm), and the experience replay times are set to 0.

[0060] Step S3.2, initialize the remaining subspace and the set of possible actions to obtain the initial subspace set (i.e., the entire palletizing space) and the initial set of possible actions. The remaining subspace is a two-dimensional space that can be placed in a top-down state, represented by the coordinates of the lower left corner and the upper right corner (such as the remaining subspace after initialization is represented as (0, 0, 1610, 1610), the first two digits are the coordinates of the lower left corner, and the last two digits are the coordinates of the upper right corner); the action is defined as a set of information including "the model of the brick placed, the coordinates of the placement point, the coordinates of the placement point, the placement direction, and the layer identification", where the placement direction is defined as a number from 1 to 8, but not all shapes of bricks have eight allowed placement directions, and the number of directions for each shape of brick is different. The layer identification is a 0 / 1 number, which defines 0 as no layering and 1 as layering required.

[0061] For each feasible block, do the following: Step S3.2.1. Select the brick type with the largest number as the priority brick type, select the first four remaining subspaces after sorting (divide the stacking space into multiple remaining subspaces along the edges of the feasible blocks that have been placed), traverse all directions in which this type of brick can be placed, and the four corners of the four remaining subspaces, and use the constraints to determine whether it can be placed. If so, generate an action and store it in the feasible action set, otherwise skip it.

[0062] According to the placement point and the occupation of the bricks in the x-axis and y-axis directions, calculate the action distance of each action to the subspace where it is located, sort all the possible actions from small to large according to this distance, and select the first 15 possible actions as the optimal possible action set in the current state. If the action set is empty, initialize the remaining subspace (enter the lower layer, still use the stacking space as the initial subspace), set the layer flag to 1, repeat the above action collection operation, if the action set is still empty, it means that the stacking space is full, set the end flag (done) to 1.

[0063] Step S3.2.2: All 15 feasible actions in the optimal feasible action set and the current state can be input into the dual deep Q network in the form of state-action pairs to obtain 15 Q values, and the action corresponding to the maximum Q value is selected as the selected action.

[0064] Step S3.2.3, execute the drop operation, that is, place the corresponding brick into the loading space in a specific direction and position according to the selected action.

[0065] Based on the information in the selected action, the current height map is updated to obtain u_plain and l_plain after placing the specified brick type at the specified point in the specified direction. Update the remaining subspace set, delete the remaining subspaces that intersect with the area occupied by the feasible block, use the edge of the feasible block as the dividing line, divide the deleted remaining subspaces to obtain new remaining subspaces and store them in the remaining subspace set, determine whether the newly added subspace is included in the remaining subspaces, and delete the included space. According to the Manhattan distance, take the minimum value of the distance between the subspace and each bottom corner of the stacking space as the standard, sort the remaining subspaces from small to large by distance, and obtain a new set of remaining subspaces. Increase the training step by one, and reduce the number of brick types placed in the data dictionary by one.

[0066] Step S3.2.4, calculate reward: calculate the reward after performing the current action in the current state according to the reward mechanism.

[0067] Step S3.2.5: Perform action generation based on the updated state and the remaining subspace set to obtain the next set of available actions. Pack the state before update, the updated state, the selected action, the next set of actions, the reward, and the end flag into the experience pool. If the end flag (done) is 1, that is, the current palletizing space is full, the experience accumulation of this round is ended, and step S3.3 is performed. Otherwise, return to step S3.2.1.

[0068] Step S3.3, perform experience replay (learning process), randomly extract 32 pieces of experience data (stacked data) from the experience pool, input the state and action into the main network (model) to obtain the Q value, set the model to evaluation mode, input the next state and the next set of possible actions into the main network (model) and the target network (target_model) to obtain the next_q value (output of the main network) and next_target value (output of the target network), modify the model network to training mode, and calculate the target value target. The formula is as follows: in, for next_q, for next_target, is the current state, For the next state, For the current action, For actionable actions, For the current reward, is the learning rate.

[0069] Use the mean square error to calculate the loss function of Q and target, clear the gradient of the model optimizer, use the calculated loss to perform gradient reversal to update the network parameters in the main network (model), and the number of experience replays + 1. After every ten times of experience replay, copy the parameters in the model network to the target network (target_model). Repeat this step for 10 rounds. After the experience replay is finished, save the target network (target_model) and return to step S3 for the next round of model training.

[0070] Finally, the brick stacking model is obtained according to the network parameters in the target network (target_model).

[0071] The search and palletizing phase is described in detail below.

[0072] Step S1, obtain the list of bricks to be stacked and initialize feasible blocks.

[0073] Step S2, initialize the state (state), end flag (done), and height map (u_plain and l_plain) in the environment, read the trained brick stacking model, process each feasible block determined in the list of bricks to be stacked based on the brick stacking model, determine the corresponding action, and stack them in the stacking space.

[0074] The specific process is similar to steps S3.2.1, S3.2.2, and S3.2.3 in the model training phase. The step accumulation operation is omitted in S3.2.3. The action can also be determined in combination with the greedy strategy.

[0075] Step S3, draw a three-dimensional (3D) image, and update the image synchronously for each drop operation to represent the real-time situation of the pallet shape in the palletizing space. When the end flag (done) is 1, the palletizing is ended, and the space occupancy rate of the bricks and the final three-dimensional diagram of the pallet shape are output, and the palletizing is completed.

[0076] Figure 4 This is the second flow diagram of the method for stacking highly heterogeneous bricks provided by the present invention, such as Figure 4 As shown in the figure, the specific implementation process of the palletizing operation in the model training phase and the palletizing search phase.

[0077] The strongly heterogeneous brick stacking method provided by the embodiment of the present invention obtains a brick stacking model by reinforcement learning of a dual deep Q network, which can replace manual decision-making, learn the optimal strategy and determine the action by interacting with the environment in a dynamic environment, and can perform brick stacking quickly and efficiently, thereby improving the efficiency of brick stacking.

[0078] In some embodiments, determining a set of feasible actions corresponding to a current training state and a current sample feasible block in a sample brick list based on the space occupancy information in the palletizing space includes: Based on the space occupancy information in the palletizing space, determining the palletized feasible blocks in the palletizing space; Based on the edges of the feasible blocks that have been stacked, the remaining space in the stacking space is divided to determine a plurality of remaining subspaces; sorting the plurality of residual subspaces based on the Manhattan distance between each residual subspace and the palletizing space, and determining a plurality of optimal residual subspaces; Based on the placement direction of the current sample feasible block, the four corners of multiple optimal residual subspaces are traversed to determine the feasible action set corresponding to the current sample feasible block.

[0079] Specifically, the steps of determining a feasible action set corresponding to a feasible block are described in detail below.

[0080] First, construct a general block and determine the current feasible block set. A general block is not a single block of regular shape. It can be composed of multiple feasible blocks of different shapes and specifications, and there may be inevitable space waste in the middle. Take the full length and full width of the stacking space as the construction condition of the general block, and combine and splice the independent feasible blocks in different ways to obtain the current feasible block set that meets the constraints. The upper limit of the number of feasible blocks in the set can be set not to exceed the preset number.

[0081] Secondly, the generation and update rules of the remaining subspace are defined, and the palletizing space is divided into several largest remaining subspaces along the edges of the placed feasible blocks.

[0082] Each time an action is generated, the optimal remaining subspace is first screened. After sorting the current remaining subspace set according to the Manhattan distance (the distance between the remaining subspace and the entire stacking space), the m remaining subspaces with the smallest distance are screened out to form the current optimal remaining subspace set C. After each feasible block placement action is completed, the entire remaining subspace set is re-divided and updated, unavailable space is removed (all remaining brick types are traversed and judged. If the length and width of the space are not enough to accommodate any placement direction of any brick type, it means that the space is unavailable), and the included space is removed (space inclusion may occur during the space division process).

[0083] The update of the remaining subspace needs to be synchronized after each action is completed.

[0084] Again, it is stipulated that the placement rule of the feasible block is to be placed only at the four corners of the remaining subspace each time. Adopting the idea of hierarchical placement, it is stipulated that the coordinates of the feasible block are represented by the coordinates of the lower left rear corner. It is stipulated that the lower left rear corner of the initial palletizing space is the coordinate origin, and a three-dimensional rectangular coordinate system is established with the length as the x-axis, the width as the y-axis, and the height as the z-axis.

[0085] Finally, generate the set of feasible actions. The set composed of the size and brick type of the placed feasible block, the placement point, the placement direction, and the layer-changing placement identifier represents the action. The layer-changing identifier is a 0 / 1 constant. When the set of feasible actions for the current layer is empty, it means the current layer is full, and the layer-changing identifier written into the action is set to 1. When the layer-changing identifier is 1 and the set of feasible actions for the next step is empty, it means the palletizing space is full and the boxing is completed.

[0086] Traverse all the feasible blocks b in the current set of feasible blocks and all m directions orthogonal to the coordinate axes of the current optimal remaining subspace, and sequentially judge whether the specified constraint conditions are met, eliminate the cases that do not meet the conditions, generate the set of feasible actions A, and the number of actions is less than or equal to 4 * b * m * d (d is the number of possible placement directions). According to the action distance between the placed feasible block and the corresponding subspace, sort the set of feasible actions, and select the u actions with the smallest distance to form the current optimal set of feasible actions A'.

[0087] The action distance can be understood as follows: when placing bricks in each subspace, according to the rules, the four corners will be selected, and the distances from the two side faces of the subspace farthest from the brick will be calculated for each corner to form the coordinates (d2, d1). If the action distance of action a is (d2, d1) and the action distance of action b is (c1, c2), then when comparing these two actions, first compare d2 with c1. If d2 < c1, the action distance of action a is less than that of action b. If d2 = c1, then compare d1 with c2. If d1 < c1, the action distance of action a is less than that of action b.

[0088] The constraint conditions can include: (1) Boundary constraint: The placed bricks cannot exceed the space specified by the kiln loading cart to ensure that all bricks on the kiln loading cart are completely accommodated after palletizing; (2) Non-overlap constraint: The spaces occupied by the bricks placed on the kiln loading cart cannot overlap with each other; (3) Direction constraint: When the bricks are stacked on the kiln loading cart, they should meet several specified placement directions, and the directions are all orthogonal to the x-axis, y-axis, and z-axis of the loading space; (4) Full support constraint: When the bricks are palletized, the bottom of each brick cannot be suspended, that is, the bottom of each brick should be in contact with the top surface of at least one adjacent brick or directly in contact with the bottom surface of the kiln loading cart.

[0089] When constructing general blocks, in order to satisfy the full support constraint, the constructed general blocks need to be in the shape of several columns or rows of the same height and compactly spliced.

[0090] The strongly heterogeneous brick stacking method provided by the embodiment of the present invention determines the set of feasible actions corresponding to the current training state and the current sample feasible block in the sample brick list according to the space occupancy information in the stacking space, and learns the optimal strategy and determines the action through interaction with the environment in a dynamic environment, so as to quickly and efficiently stack bricks.

[0091] In some embodiments, after determining the current training state and the feasible action set corresponding to the current sample feasible block in the sample brick list based on the space occupancy information in the palletizing space, the method further includes: Based on the greedy strategy, the current training action under the current training state is randomly selected from the set of feasible actions corresponding to the feasible block of the current sample.

[0092] Specifically, in step S3.2.2 of the model training phase, the greedy strategy ( ) Select the best action from the set of best possible actions.

[0093] set up is the coefficient that decays from 1 to 0 according to the following formula: in, The value is 0, The value is 1, is the training step length, The value is set to 3000 (that is, when the training step length reaches 3000, The rate of decrease reaches its maximum value).

[0094] In step S3.2.2, a random decimal between 0 and 1 is selected. When , an action is randomly selected from the optimal feasible action set as the action taken in the current state. Conversely, all 15 actions in the current optimal feasible action set and the current state are input into the neural network in the form of state-action pairs to obtain 15 Q values, and the action corresponding to the maximum Q value is selected as the selected action.

[0095] In the search for palletizing phase, you can set is a constant of 0 (no more decay).

[0096] The strongly heterogeneous brick stacking method provided in the embodiment of the present invention uses a greedy strategy to determine the current training action, and can obtain the global optimal solution through local optimal selection, or quickly find a better approximate solution when there is no global optimal solution, thereby improving the training speed and efficiency of the model.

[0097] In some embodiments, the reward is determined based on the following steps: Based on the space occupancy information of the palletizing space, determine the current space occupancy rate, the volume occupancy rate at the current maximum occupancy height, and the average height of the currently occupied space; Determine the reward corresponding to the current action based on the current space occupancy, the volume occupancy at the current maximum occupied height, and the average height of the current occupied space.

[0098] Specifically, the rewards can be set as follows: in, is the reward corresponding to the current action, It indicates the ratio of the volume of bricks stacked in the palletizing space to the volume of the loading space, i.e. the space occupancy rate; Indicates the volume occupancy rate at the current maximum occupied height; is the average height of the occupied space. , and is the weight coefficient, .

[0099] The current space occupancy rate, the volume occupancy rate at the current maximum occupied height, and the average height of the currently occupied space can all be determined based on the space occupancy information of the palletizing space.

[0100] It is the space occupancy rate, which implies the condition of giving priority to feasible blocks with larger volume.

[0101] It represents the volume occupancy rate at the current maximum occupied height, and the formula is as follows: in, is the sequence number of the feasible block (brick) currently placed, is the number of feasible blocks that have been placed, is the volume of the feasible block that has been placed, is the length of the palletizing space, is the width of the palletizing space, is the maximum height currently occupied, The closer it is to 1, the more compact the current stack is.

[0102] Defined as the average height of the occupied space, it is calculated as follows: in, is the bottom area of ​​the palletizing space, and the height of each point (x, y) within this range is , the maximum height is , The closer it is to 1, the closer the stacks are to the same height, which not only constrains the compactness of the stacks, but also improves the stability of the stacks.

[0103] The highly heterogeneous brick stacking method provided in the embodiment of the present invention determines the reward corresponding to the current action according to the current space occupancy rate, the volume occupancy rate at the current maximum occupied height and the average height of the currently occupied space, thereby improving the stability of the stacking type.

[0104] In some embodiments, the feasible blocks in the list of bricks to be stacked are determined based on the following steps: Process the list of bricks to be stacked and determine multiple brick types; Generate two regular cubic bricks corresponding to each brick type by taking the upper bottom surface and the lower bottom surface of each brick type as the bottom surface; Based on the two regular cubic bricks corresponding to each brick type, the feasible blocks in the list of bricks to be stacked are determined.

[0105] Specifically, bricks produced in factories often have strong heterogeneity problems due to different models, and their shapes are not limited to cubes.

[0106] Data processing can be performed on the list of bricks to be stacked to determine multiple brick types and obtain a data set containing only brick type, size, and quantity. Each brick can be taken as the bottom surface and the height as the bottom surface to generate two regular cubic bricks, thereby obtaining a data set containing brick type, upper bottom surface cube size, lower bottom surface cube size, and quantity, and then determining feasible blocks.

[0107] Since bricks are highly heterogeneous and have the characteristic that "all sides of bricks are flat", and the upper and lower bottom surfaces must be parallel to each other when placed, when a brick is at the same height, whether it can be placed in the current space while minimizing space waste can be determined by the criterion that after the brick is placed in the space, its upper and lower surfaces intersect with the upper and lower surfaces of the existing bricks in the space respectively. Therefore, this method uses upper bottom surface cubes and lower bottom surface cubes to fully characterize a brick.

[0108] The highly heterogeneous brick stacking method provided by the embodiment of the present invention uses the upper and lower bottom surfaces of each brick as the bottom surface and its own height as the height to generate two regular cubic bricks, generate feasible blocks, and solve the problem of space waste caused by mixed loading of highly heterogeneous standard cubic bricks, parallelogram bricks, isosceles trapezoidal bricks, and right-angled trapezoidal bricks.

[0109] In some embodiments, the model structures of the main network and the target network are the same; the main network includes a feature extraction module, a feature concatenation module, and a feature recognition module; The feature extraction module includes a convolution submodule and a fully connected submodule connected in sequence; the convolution submodule includes a plurality of convolution layers connected in sequence; the fully connected submodule includes a plurality of fully connected layers connected in sequence; the feature recognition module includes a plurality of fully connected layers connected in sequence; The input of the feature extraction module is the current state; the input of the feature splicing module is the output of the feature extraction module and the current action; the input of the feature recognition module is the output of the feature splicing module.

[0110] Specifically, the model structures of the main network and the target network of the dual deep Q-network are the same. Figure 5 It is a model structure diagram of the dual deep Q network provided by the present invention, such as Figure 5 As shown, the main network includes a feature extraction module, a feature splicing module and a feature recognition module.

[0111] The feature extraction module includes a convolution submodule and a fully connected submodule connected in sequence. The convolution submodule includes multiple convolution layers connected in sequence, such as 5 convolution layers; the fully connected submodule includes multiple fully connected layers connected in sequence, such as 3 fully connected layers. The feature recognition module includes multiple fully connected layers connected in sequence, such as 2 fully connected layers.

[0112] The input of the feature extraction module is the current state, which is used to extract features and reduce the dimension of the two-dimensional height map. The input of the feature splicing module is the output of the feature extraction module and the current action (one-dimensional), which is used to splice two features. The input of the feature recognition module is the output of the feature splicing module, which is used to determine the Q value based on the spliced ​​features.

[0113] The highly heterogeneous brick stacking method provided by the embodiment of the present invention has a main network including a feature extraction module, a feature splicing module and a feature recognition module, which accelerates the speed of the algorithm converging to the optimal solution, avoids the emergence of local optimal solutions, and effectively maximizes the space occupancy rate.

[0114] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below and the method described above can refer to each other.

[0115] Figure 6 Schematic diagram of the structure of the device for stacking highly heterogeneous bricks provided by the present invention. Figure 6 As shown, the device comprises: A state determination unit 610 is used to stack the current feasible blocks in the list of brick materials to be stacked into the stacking space of the target kiln loading vehicle based on the current action, determine the space occupancy information of the stacking space, and determine the current state based on the space occupancy information; The action evaluation unit 620 is used to input the current state and the current action into the brick stacking model to obtain the action value of each possible action output by the brick stacking model; An action determination unit 630 is used to determine the next action based on the action value of each feasible action, and to stack the next feasible block in the list of brick materials to be stacked into the stacking space based on the next action; Among them, the brick stacking model is obtained through reinforcement learning of the dual deep Q network based on the sample brick list, the sample actions corresponding to each feasible block in the sample brick list, and the rewards corresponding to the sample actions; the rewards are determined based on the spatial occupancy information of the stacking space.

[0116] The strongly heterogeneous brick stacking device provided by the embodiment of the present invention stacks the current feasible blocks in the list of bricks to be stacked into the stacking space of the target kiln loading vehicle according to the current action, determines the space occupancy information of the stacking space, and determines the current state according to the space occupancy information; inputs the current state and the current action into the brick stacking model to obtain the action value of each feasible action output by the brick stacking model; determines the next action according to the action value of each feasible action, and stacks the next feasible block in the list of bricks to be stacked into the stacking space according to the next action; because the brick stacking model is obtained by reinforcement learning of the dual deep Q network based on the sample brick list, the sample actions corresponding to each feasible block in the sample brick list and the rewards corresponding to the sample actions, it can replace manual decision-making, learn the optimal strategy through interaction with the environment in a dynamic environment, determine the action, and can quickly and efficiently stack bricks, thereby improving the efficiency of brick stacking and improving the space utilization of the kiln loading vehicle.

[0117] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 710, a communication interface (Communications Interface) 720, a memory (Memory) 730 and a communication bus (Communications Bus) 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic command in the memory 730 to execute the method described in the above embodiment, for example: Based on the current action, the current feasible block in the list of brick materials to be stacked is stacked into the stacking space of the target kiln loading car, the space occupancy information of the stacking space is determined, and the current state is determined based on the space occupancy information; the current state and the current action are input into the brick stacking model to obtain the action value of each feasible action output by the brick stacking model; the next action is determined based on the action value of each feasible action, and based on the next action, the next feasible block in the list of brick materials to be stacked is stacked into the stacking space; wherein, the brick stacking model is based on the sample brick material list, the sample actions corresponding to each feasible block in the sample brick material list and the rewards corresponding to the sample actions, and is obtained through reinforcement learning of the dual deep Q network; the reward is determined based on the space occupancy information of the stacking space.

[0118] In addition, the logic commands in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0119] The processor in the electronic device provided in the embodiment of the present invention can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0120] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0121] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0122] An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described above is implemented.

[0123] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for stacking highly heterogeneous bricks, characterized in that: include: Based on the current action, currently feasible blocks in the list of brick materials to be stacked are stacked into the stacking space of the target kiln loading vehicle, space occupancy information of the stacking space is determined, and the current state is determined based on the space occupancy information; Input the current state and the current action into a brick stacking model to obtain the action value of each possible action output by the brick stacking model; Determine the next action based on the action value of each feasible action, and stack the next feasible block in the list of bricks to be stacked into the stacking space based on the next action; Among them, the brick stacking model is obtained after reinforcement learning of a dual deep Q network based on a sample brick list, sample actions corresponding to each feasible block in the sample brick list and rewards corresponding to the sample actions; the reward is determined based on the space occupancy information of the stacking space.

2. The method for stacking highly heterogeneous bricks according to claim 1, characterized in that: The brick stacking model is trained based on the following steps: Determine a set of feasible actions corresponding to a current training state and a current sample feasible block in the sample brick material list based on the space occupancy information in the palletizing space; Inputting the current training state and each actionable action in the action set into a main network in the dual deep Q network to obtain an action value of each actionable action output by the main network; Determining a current training action in the current training state based on the action value of each possible action; Based on the current training action, the current sample feasible block is stacked in the stacking space, the space occupancy information in the stacking space is updated, and the next training state and the reward corresponding to the current training action are determined; Determine the experience data corresponding to the current training state based on the current training state, the next training state, the current training action, the set of feasible actions corresponding to the current sample feasible block, and the reward corresponding to the current training action; Training the main network based on empirical data corresponding to a plurality of training states, and updating model parameters of the target network in the dual deep Q network based on the model parameters of the main network; The brick stacking model is determined based on the target network obtained after the model training phase.

3. The method for stacking highly heterogeneous bricks according to claim 2, characterized in that: The determining of the current training state and the feasible action set corresponding to the current sample feasible block in the sample brick material list based on the space occupancy information in the palletizing space includes: Based on the space occupancy information in the stacking space, determining the feasible blocks that have been stacked in the stacking space; Based on the edges of the stacked feasible blocks, the remaining space in the stacking space is divided to determine a plurality of remaining subspaces; sorting the plurality of remaining subspaces based on the Manhattan distance between each remaining subspace and the palletizing space to determine a plurality of optimal remaining subspaces; The four corners of the plurality of optimal remaining subspaces are traversed based on the placement direction of the current sample feasible block to determine a feasible action set corresponding to the current sample feasible block.

4. The method for stacking highly heterogeneous bricks according to claim 2, characterized in that: After determining the current training state and the feasible action set corresponding to the current sample feasible block in the sample brick material list based on the space occupancy information in the palletizing space, the method further includes: The current training action in the current training state is randomly selected from the feasible action set corresponding to the current sample feasible block based on a greedy strategy.

5. The method for stacking highly heterogeneous bricks according to claim 1, characterized in that: The rewards are determined based on the following steps: Based on the space occupancy information of the palletizing space, determine the current space occupancy rate, the volume occupancy rate at the current maximum occupancy height, and the average height of the currently occupied space; A reward corresponding to a current action is determined based on the current space occupancy rate, the volume occupancy rate at the current maximum occupied height, and the average height of the current occupied space.

6. The method for stacking highly heterogeneous bricks according to claim 1, characterized in that: The feasible blocks in the list of bricks to be stacked are determined based on the following steps: Processing the list of bricks to be stacked to determine multiple brick types; Generate two regular cubic bricks corresponding to each brick type by taking the upper bottom surface and the lower bottom surface of each brick type as the bottom surface; Based on the two regular cubic bricks corresponding to each brick type, feasible blocks in the list of bricks to be stacked are determined.

7. The method for stacking highly heterogeneous bricks according to claim 2, characterized in that: The main network and the target network have the same model structure; the main network includes a feature extraction module, a feature splicing module and a feature recognition module; The feature extraction module includes a convolution submodule and a fully connected submodule connected in sequence; the convolution submodule includes a plurality of convolution layers connected in sequence; the fully connected submodule includes a plurality of fully connected layers connected in sequence; the feature recognition module includes a plurality of fully connected layers connected in sequence; The input of the feature extraction module is the current state; the input of the feature splicing module is the output of the feature extraction module and the current action; the input of the feature recognition module is the output of the feature splicing module.

8. A device for stacking highly heterogeneous bricks, characterized in that: include: A state determination unit, for stacking the currently feasible blocks in the list of brick materials to be stacked into the stacking space of the target kiln loading vehicle based on the current action, determining the space occupancy information of the stacking space, and determining the current state based on the space occupancy information; An action evaluation unit, used for inputting the current state and the current action into a brick stacking model to obtain the action value of each possible action output by the brick stacking model; An action determination unit, used for determining a next action based on the action value of each feasible action, and stacking the next feasible block in the list of brick materials to be stacked into the stacking space based on the next action; Among them, the brick stacking model is obtained after reinforcement learning of a dual deep Q network based on a sample brick list, sample actions corresponding to each feasible block in the sample brick list and rewards corresponding to the sample actions; the reward is determined based on the space occupancy information of the stacking space.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the strongly heterogeneous brick stacking method described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for stacking highly heterogeneous bricks as described in any one of claims 1 to 7 is implemented.

Images