Intelligent decision-making and online reinforcement learning method for two-leg support shifting action

Abstract

The application discloses a two-column hydraulic support shifting action intelligent decision and online reinforcement learning method, relates to the technical field of support shifting action decision, and comprises the following steps: firstly, according to the geological conditions of a working face, the action sequence executed in a single support shifting process and a neighboring support collaborative shifting process is determined, and the value of each data item in the geological conditions is subjected to fuzzy attribute definition; then, a neighboring support collaborative priori knowledge table and a single support shifting process priori knowledge table are constructed; then, the online learning process is completed according to the reward value generated by the created experience pool; and finally, the support shifting action decision is executed. The application can efficiently make a shifting strategy according to different geological conditions, lays a good foundation for subsequent coal mining of a coal mining machine, and further improves the working efficiency.

Description

Technical Field

[0001] This invention belongs to the field of decision-making technology for scaffold movement, specifically relating to an intelligent decision-making and online reinforcement learning method for the movement of a two-column scaffold. Background Technology

[0002] Intelligent equipment in fully mechanized coal mining faces using hydraulic supports is key to intelligent coal mining. The core of this system is the coordinated operation of the coal mining machine, hydraulic supports, and scraper conveyors. Continuous improvement in intelligent fully mechanized mining can enhance the efficiency and quality of coal mining operations, prevent safety accidents, ensure production safety, and reduce production costs. The complex environment of fully mechanized coal mining faces, including the degree of roof fragmentation, roof integrity, and floor flatness, all influence the selection of hydraulic support movement actions. These movement actions, in turn, affect the coordinated control with the coal mining machine and the machine's mining efficiency. Intelligent coal mining is inseparable from intelligent hydraulic supports. Hydraulic support movement is a crucial step in the coal mining process. Hydraulic supports need to efficiently determine movement strategies based on different geological conditions to lay a solid foundation for subsequent coal mining operations.

[0003] Currently, research on the movement judgment of hydraulic supports mainly relies on the position of the coal mining machine to select the appropriate action. Examples include: Tian Chengjin et al.'s 2012 research on automated follow-up technology based on the SAC type hydraulic support electro-hydraulic control system; Fan Jingdao et al.'s 2016 research on the integrated design and practice of the Huangling intelligent unmanned mining face system; and Lei Zhaoyuan et al.'s 2019 research on automatic follow-up control technology of hydraulic supports in high-extraction intelligent mining faces. In fully mechanized mining faces, the movement of hydraulic supports lags slightly behind that of the coal mining machine. The above research on hydraulic support movement primarily focuses on judging the movement based on the position of the coal mining machine, while research specifically on intelligent decision-making for hydraulic support movement is almost nonexistent.

[0004] Therefore, the key is to make intelligent decisions on the relocation of hydraulic supports based on known geological conditions. Summary of the Invention

[0005] The existing hydraulic support motion judgment methods suffer from the limitation that the hydraulic support's motion lags slightly behind that of the coal mining machine. Previous studies on hydraulic support motion primarily rely on the position of the coal mining machine for on-the-fly judgment, while research on intelligent decision-making for hydraulic support movement is almost nonexistent. This invention provides an intelligent decision-making and online reinforcement learning method for two-column support movement. Based on practical considerations, the intelligent decision-making for hydraulic support movement is divided into two main categories. Furthermore, it innovatively proposes utilizing an experience pool for online learning and applying reinforcement learning to the intelligent decision-making of hydraulic support movement, thereby further facilitating the selection of hydraulic support movement actions.

[0006] The solution adopted by this invention to solve its technical problem is: an intelligent decision-making and online reinforcement learning method for the movement of a two-column support, comprising the following steps:

[0007] Step 1: Based on the geological conditions of the working face, determine the sequence of actions to be performed during the single-support relocation process and the coordinated relocation process of adjacent supports;

[0008] Step 2: Determine the geological conditions of the working face and define fuzzy attributes for the values ​​of each data item in the geological conditions;

[0009] Step 3: Construct a prior knowledge table of adjacent frame collaboration based on push-pull collaboration and bottom-adjustment collaboration, and create the collaboration probability of the two under different working states respectively;

[0010] Step 4: Construct a prior knowledge table for the single-unit support frame relocation process;

[0011] Step 5: Construct a support shifting experience pool and record each manual selection during the working face back mining process; including the push conveyor coordination experience pool, the bottom adjustment coordination experience pool, and the single support shifting experience pool;

[0012] Step Six: Based on the reward values ​​generated from the experience data in the experience pool in Step Five, perform an online learning process on the prior knowledge of push-slide coordination, bottom adjustment coordination, and single-support action to make real-time corrections;

[0013] Step 7: Execute the decision to move the support frame; including the following steps:

[0014] S701. Obtain the fuzzy variable values ​​of the geological environment state of the working face, and set the "scraper conveyor adhesion" to search for state variables. "Working face inclination angle" to find state variables Search for state variables such as "top plate fracture degree" and "bottom plate hardness".

[0015] S702. Based on the state variables read in S701, perform online learning and updating correction of the complete prior probability table according to step six.

[0016] S703. Based on the update result of S703, select different moving action decisions in a probabilistic manner.

[0017] Furthermore, the single-support relocation process in step one includes a conventional relocation process. Pressurized frame transfer process Harmony and Balance Frame The adjacent frame coordinated moving process includes adjacent frames moving together without pushing or sliding. Neighboring frames push and slide in coordination to move the frames Neighboring frames without bottom adjustment and coordinated moving. Coordinated movement of the frame with the adjacent frame.

[0018] Furthermore, the conventional moving action sequence is as follows: lower the column - extend the bottom lifting jack - retract the pushing jack - raise the column;

[0019] The pressurized frame relocation action sequence includes: first, slightly lowering the column, extending the bottom jack, and retracting the push jack, and repeating this sequence multiple times; until the push jack is completely retracted, then raising the column;

[0020] The balancing and shifting action sequence includes:

[0021] Phase 1: Slightly lower the column - raise the bottom - move the jack;

[0022] Phase Two: Balance Adjustment - Lowering the Column - Raising the Base - Retracting the Push Jack - Raising the Column.

[0023] Furthermore, the geological conditions of the working face in step one include the working inclination angle, the adhesion of the scraper conveyor, the degree of crushing of the top plate, the flatness of the top plate, and the flatness of the bottom plate; the working face inclination angle and the adhesion of the scraper conveyor determine whether the adjacent frames participate in the coordinated frame movement; the degree of crushing of the top plate, the flatness of the top plate, and the flatness of the bottom plate determine the current frame movement process.

[0024] Furthermore, the definition of fuzzy attribute values ​​in step one includes:

[0025] Working tilt angle values: small, relatively small, relatively large, large;

[0026] The adhesion values ​​for scraper conveyors are: small, relatively small, relatively large, and large.

[0027] The degree of breakage of the roof slab can be categorized as: intact, slightly broken, or broken.

[0028] The hardness values ​​for the base plate are: soft, relatively soft, relatively hard, and hard.

[0029] Furthermore, the correction process in step six includes:

[0030] S1. First, calculate the difference between the experience pool probability and the prior probability using formula (1):

[0031]

[0032] And if the difference satisfies Then update the prior probability;

[0033] S2. Introduce the reward function value generated by the experience pool into the prior probability, and calculate the generated probability using formula (2).

[0034]

[0035] S3. Update the prior probabilities of each state in the information category using formula (3);

[0036]

[0037] Where K represents the information category, and i represents the prior probability of the i-th state in information category K.

[0038] Furthermore, the method of selecting the movement action in step seven includes:

[0039] 1) Push-slide coordinated action decision-making:

[0040] (a) Generate random numbers θ∈(0,1)

[0041] (b) If Then select the action of moving the adjacent frame without pushing or sliding in a coordinated manner. Otherwise, choose to move the adjacent frame in a coordinated manner.

[0042] 2) Decision-making based on underlying coordination:

[0043] (a) Generate random numbers θ∈(0,1)

[0044] (b) If Then select the action of moving the adjacent frame without bottom adjustment in a coordinated manner. Otherwise, choose to coordinate the movement of the adjacent frame.

[0045] 3) Single-frame action decision-making:

[0046] (a) Generate random numbers θ∈(0,1)

[0047] (b)

[0048] like Then choose the conventional moving process.

[0049] Otherwise if Then select the pressurized frame transfer process. Otherwise, choose to adjust the balance and move the frame.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] This invention provides an intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support. It offers two main categories of movement processes: single-support movement and adjacent-support collaborative movement. Furthermore, it innovatively proposes utilizing an experience pool for online learning, applying reinforcement learning to the intelligent decision-making process for hydraulic support movement. This enables the hydraulic support to efficiently determine movement strategies based on different geological conditions. Based on reinforcement learning, and employing an online learning approach, it completes the action selection for hydraulic support movement, thereby further ensuring production safety and improving work efficiency. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the working surface support shifting process of the present invention.

[0053] In the diagram: 1. Roof plate; 2. Bottom plate; 3. Coal wall; 4. Scraper conveyor; 5. Pushing jack; 6. Bottom lifting jack; 7. Bottom adjusting jack; 8. Balancing jack; 9. Column; 10. Telescopic beam; 11. Side guard plate. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] Please see Figure 1 This invention provides a technical solution for intelligent decision-making and online reinforcement learning for the movement of a two-column hydraulic support. Based on the actual situation, the intelligent decision-making for the movement of the hydraulic support is divided into two main categories. It also innovatively proposes to use an experience pool for online learning and apply reinforcement learning to the intelligent decision-making of the movement of the hydraulic support, thereby further facilitating the selection of the movement for the hydraulic support.

[0056] Example 1:

[0057] See Figure 1 The movement of the hydraulic support refers to the movement of the hydraulic support towards the coal face 3. This embodiment provides an intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support, including the following steps:

[0058] Step 1: Based on the geological conditions of the working face, determine the sequence of actions for the single-support relocation process and the coordinated relocation process of adjacent supports; the single-support relocation process includes the conventional relocation process. Pressurized frame transfer process Adjusting the balance and moving the frame and definition, specifically:

[0059] (1) Conventional moving process The sequence of actions to be performed includes:

[0060] Lower the column - extend the bottom jack - retract the push jack - raise the column;

[0061] (2) Pressure-driven frame transfer process The sequence of actions to be performed includes:

[0062] First, execute the sub-sequence of slightly lowering the column, extending the bottom jack, and retracting the push jack, and repeat this operation multiple times; until the push jack is completely retracted, then raise the column.

[0063] (3) Adjusting the balance and moving the frame The action sequence includes:

[0064] Phase 1: Slightly lower the column - raise the bottom - move the jack;

[0065] Phase Two: Balance Adjustment - Lowering the Column - Raising the Base - Retracting the Push Jack - Raising the Column.

[0066] The process of coordinated relocation of adjacent supports refers to the process by which the current support selects adjacent supports to participate in the coordinated relocation based on geological conditions, including:

[0067] (1) Neighboring frames do not push and slide in coordination to move the frames.

[0068] The current support system is selected to perform the above-mentioned single support relocation process based on geological conditions;

[0069] (2) Collaborative movement of adjacent frames by pushing and sliding

[0070] Adjacent support actions: extend and push the jacks; and the current support selects the above-mentioned single support relocation process according to geological conditions.

[0071] (3) Collaborative relocation of adjacent racks without bottom adjustment

[0072] Adjacent support actions: extend the bottom adjustment jacks; and the current support selects to perform the above-mentioned single support relocation process according to geological conditions;

[0073] (4) Coordinated movement of adjacent frames with bottom adjustment

[0074] Adjacent support actions: extend the bottom adjustment jack and simultaneously extend the push-moving jack; and the current support selects to perform the above single support relocation process according to geological conditions.

[0075] Step 2: Determine the geological conditions of the working face and define fuzzy attributes for each data item within these conditions. The geological conditions include the working face inclination angle, scraper conveyor adhesion, roof fragmentation, roof flatness, and floor flatness. The working face inclination angle and scraper conveyor adhesion determine whether adjacent supports participate in coordinated support relocation, while the roof fragmentation, roof flatness, and floor flatness determine the current support relocation process. Fuzzy attribute values ​​are then defined for each of these data items.

[0076] Working tilt angle values: small, relatively small, relatively large, large;

[0077] The adhesion values ​​for scraper conveyors are: small, relatively small, relatively large, and large.

[0078] The degree of breakage of the roof slab can be categorized as: intact, slightly broken, or broken.

[0079] The hardness values ​​for the base plate are: soft, relatively soft, relatively hard, and hard.

[0080] Step 3: Based on the experience of coal mining experts and front-line operators, establish a prior knowledge table for adjacent frame coordination of push conveyor and bottom adjustment coordination:

[0081] Table 1: Prior Knowledge Table of Push-Slide Collaboration

[0082]

[0083] Table 2: Prior Knowledge Table of Coordination with Baseline Tone

[0084]

[0085]

[0086] Step 4: Construct a prior knowledge table for the single-unit support frame relocation process:

[0087] Table 3: Prior Knowledge for Single Support Frame Relocation Process

[0088]

[0089] Step 5: Establish a training pool for moving the support frame, and record each manual selection during the face recovery process; including:

[0090] (1) Push-and-slide collaborative experience pool;

[0091]

[0092]

[0093] (2) Bottom-line collaborative experience pool;

[0094]

[0095] (3) Experience pool for moving a single support frame;

[0096]

[0097]

[0098] Step Six: Based on the experience data in the experience pool from Step Five above, perform online learning, including:

[0099] 1) Statistics on the probability of actions in the experience pool

[0100]

[0101] In the formula, k = 0, 1, 2 represent the information numbers for single support, push-slide coordination, and bottom adjustment coordination, respectively; i is the state number in each type of information; and q represents the action number. For example, to calculate the action state of a single support... In China, conventional scaffolding relocation The probability is:

[0102]

[0103] In the formula Take the experience pool of moving a single support frame respectively

[0104] 2) Definition of the return value function

[0105]

[0106] In the formula, N k This represents the number of actions in the k-th type of information. N1 = 2 for push-pull coordination, N2 = 2 for bottom-tuning coordination, and N0 = 3 for support action alone. λ > 0 is the reward constant, set empirically; the set value generally makes...

[0107] 3) Online learning process

[0108] Online learning involves real-time correction of prior knowledge regarding push-slide coordination, bottom-adjustment coordination, and single-support actions based on the reward values ​​generated from the experience pool. The correction process includes:

[0109] S1. First, calculate the difference between the experience pool probability and the prior probability using formula (1):

[0110]

[0111] And if the difference satisfies Then update the prior probability;

[0112] S2. Introduce the reward function value generated by the experience pool into the prior probability, and calculate the generated probability using formula (2).

[0113]

[0114] S3. Update the prior probabilities of each state in the information category using formula (3);

[0115]

[0116] Where K represents the information category, and i represents the prior probability of the i-th state in information category K.

[0117] Step 7: Execute the decision to move the stent, including the following steps:

[0118] S701. Obtain the fuzzy variable values ​​of the geological environment state of the working face, and set the "scraper conveyor adhesion" to search for state variables. "Working face inclination angle" to find state variables Search for state variables such as "top plate fracture degree" and "bottom plate hardness".

[0119] S702. Based on the state variables read in S701, perform online learning and updating correction of the complete prior probability table according to step six.

[0120] S703. Based on the update result of S703, select the corresponding action from the push-slide coordination, bottom adjustment coordination, and single support action table in a probabilistic manner. The selection method is as follows:

[0121] 1) Push-slide coordinated action decision-making:

[0122] (a) Generate random numbers θ∈(0,1)

[0123] (b) If Then select the action of moving the adjacent frame without pushing or sliding in a coordinated manner. Otherwise, choose to move the adjacent frame in a coordinated manner.

[0124] 2) Decision-making based on underlying coordination:

[0125] (a) Generate random numbers θ∈(0,1)

[0126] (b) If Then select the action of moving the adjacent frame without bottom adjustment in a coordinated manner. Otherwise, choose to coordinate the movement of the adjacent frame.

[0127] 3) Single-frame action decision-making:

[0128] (a) Generate random numbers θ∈(0,1)

[0129] (b) If Then choose the conventional moving process.

[0130] Otherwise if Then select the pressurized frame transfer process. Otherwise, choose to adjust the balance and move the frame.

[0131] This invention provides an intelligent decision-making and online reinforcement learning method for the movement of a two-column support. Regarding the intelligent decision-making for the movement of the hydraulic support, it is based on reinforcement learning and employs online learning to complete the action selection for hydraulic support movement. The entire process is convenient and quick, providing workers with a convenient and efficient choice, and effectively improving the efficiency of support movement.

[0132] The above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent decision-making and online reinforcement learning of the movement action of a two-column hydraulic support, characterized in that: Includes the following steps: Step 1: Based on the geological conditions of the working face, determine the sequence of actions to be performed during the single-support relocation process and the coordinated relocation process of adjacent supports; the geological conditions of the working face include the working inclination angle, the adhesion of the scraper conveyor, the degree of crushing of the top plate, the flatness of the top plate, the flatness of the bottom plate, and the softness and hardness of the bottom plate; Step 2: Determine the geological conditions of the working face and define fuzzy attributes for the values ​​of each data item in the geological conditions; Step 3: Construct a prior knowledge table of adjacent frame collaboration based on push-pull collaboration and bottom-adjustment collaboration, and create the collaboration probability of the two under different working states respectively; Step 4: Construct a prior knowledge table for the single-unit support frame relocation process; Step 5: Establish a moving frame experience pool and record each manual selection during the working face back mining process; This includes experience pools for push-pull coordination, bottom adjustment coordination, and single-unit support relocation. Step Six: Based on the reward values ​​generated from the experience data in the experience pool in Step Five, perform an online learning process on the prior knowledge of push-slide coordination, bottom adjustment coordination, and single-support action to make real-time corrections; Step 7: Execute the decision to move the support frame; including the following steps: S701. Obtain the fuzzy variable values ​​of the geological environment state of the working face, and set "scraper conveyor adhesion" to search for state variables. "Working face inclination angle" is used to find the state variable. ; Search for state variables for "top plate fracture degree" and "bottom plate hardness". ; S702. Based on the state variables read in S701, perform online learning and updating correction of the complete prior probability table according to step six. S703. Based on the update result of S702, select different moving action decisions in a probabilistic manner.

2. The intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support according to claim 1, characterized in that: The single-support relocation process in step one includes the conventional support relocation process. Pressurized frame transfer process Harmony and Balance Frame The adjacent frame collaborative moving process includes adjacent frames moving collaboratively without pushing or sliding. Collaborative movement of adjacent frames , adjacent racks without bottom adjustment and coordinated rack movement Coordinated movement of the frame with the adjacent frame. .

3. The intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support according to claim 2, characterized in that: The sequence of actions performed during the conventional frame relocation process is as follows: lower the column - extend the bottom lifting jack - retract the pushing jack - raise the column; The pressurized frame relocation process includes the following action sequence: first, slightly lowering the column, extending the bottom jack, and retracting the push jack, and repeating this sequence multiple times; until the push jack is completely retracted, the column is raised. The balancing and shifting action sequence includes: Phase 1: Slightly lower the column - raise the bottom - move the jack; Phase Two: Balance Adjustment - Lowering the Column - Raising the Base - Retracting the Push Jack - Raising the Column.

4. The intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support according to claim 1, characterized in that: The working face inclination angle and the adhesion of the scraper conveyor mentioned in step one determine whether the adjacent frames participate in the coordinated frame movement; the degree of breakage of the top plate, the flatness of the top plate, and the flatness of the bottom plate determine the current frame movement process.

5. The intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support according to claim 1, characterized in that: The definition of fuzzy attribute values ​​in step two includes: Working tilt angle values: small, relatively small, relatively large, large; The adhesion values ​​for scraper conveyors are: small, relatively small, relatively large, and large. The degree of breakage of the roof slab can be categorized as: intact, slightly broken, or broken. The hardness values ​​for the base plate are: soft, relatively soft, relatively hard, and hard.

6. The intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support according to claim 1, characterized in that: The correction process in step six includes: S1. First, calculate the difference between the experience pool probability and the prior probability using formula (1): （1） And if the difference satisfies If so, then update the prior probability; S2. Introduce the reward function value generated by the experience pool into the prior probability, and calculate the generated value using formula (2). : （2） S3. Update the prior probabilities of each state in the information category using formula (3); ； Where K represents the information category, and i represents the prior probability of the i-th state in information category K.

7. The intelligent decision-making and online reinforcement learning method for the movement of a two-column hydraulic support according to claim 1, characterized in that: The method for performing the frame-shifting action in step seven includes: 1) Push-slide coordinated action decision-making: (a) Generate random numbers ; (b) If Then select the action of moving the adjacent frame without pushing or sliding in a coordinated manner. Otherwise, choose to move the adjacent frame in a coordinated manner. ; 2) Decision-making based on underlying coordination: (a) Generate random numbers ; (b) If Then select the action of moving the adjacent frame without bottom adjustment in a coordinated manner. Otherwise, choose to coordinate the movement of the adjacent frame. ; 3) Single-frame action decision-making: (a) Generate random numbers ; (b) If Then choose the conventional moving process. ; Otherwise if Then select the pressurized frame transfer process. Otherwise, choose to adjust the balance and move the frame. .

Images