A deep reinforcement learning-based particle board target detection system and method

Abstract

The application discloses a kind of deep reinforcement learning's particle board target detection system and detection method, comprising: obtaining particle board defect sample image dataset;Define the state and action space of Agent;Design dataset loader, build the framework of deep reinforcement learning target detection model;Design target detection evaluation index and reward function;Using the well-divided target detection dataset to train deep reinforcement learning target detection model, combined with target detection evaluation index, select the best deep reinforcement learning target detection model.The application uses deep reinforcement learning, continuously morphological transformation is carried out to initial candidate rectangular frame, and the position detection of target defect is completed;The method can be used for the defect position detection task of particle board image, reduces the calculation complexity, and has strong adaptive detection ability and high detection accuracy.

Description

Technical Field

[0001] This invention relates to the fields of smart forestry and computer technology, and particularly to the fields of particleboard defect detection, deep reinforcement learning and machine vision. Specifically, it relates to a particleboard target detection system and detection method based on deep reinforcement learning. Background Technology

[0002] Particleboard is a type of engineered wood product made by hot-pressing wood chips with adhesives. It is widely used in furniture manufacturing and other fields. Quality inspection during its production process is a key application area for target detection technology. Target detection is an important research direction in machine vision, currently widely used in machinery, medicine, military, and agricultural and forestry products. Establishing comprehensive target detection models and improving the accuracy of target detection tasks are current research priorities.

[0003] Object detection algorithms have evolved from early R-CNN to later two-stage detectors like Fast R-CNN, and single-stage detectors like YOLO and SSD, with continuously improving detection performance. However, existing single-stage and two-stage detectors achieve these performance improvements at the cost of increasing network size and computational complexity, resulting in issues such as complex network structures, high computational overhead, and limited adaptability in dynamic scenes.

[0004] Reinforcement learning learns optimal strategies through continuous interaction between an agent and its environment, and its decision-making mechanism is highly similar to the human visual attention system. Introducing reinforcement learning into particleboard target detection allows the detector to dynamically adjust its search strategy based on scene changes, helping to reduce computational complexity and improve system robustness.

[0005] In summary, there is a need to provide a particleboard target detection system and method to detect the location of defects on particleboard, and to further solve the problems of high computational overhead and limited adaptability of existing target detection algorithms when used for particleboard target detection. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a particleboard target detection system and method based on deep reinforcement learning, which addresses the problems of high computational overhead and limited adaptability of existing target detection algorithms. This method uses deep reinforcement learning to continuously transform the shape of the initial candidate bounding box to complete the target defect location detection. This method can be applied to the defect location detection task of particleboard images, reduces computational complexity, can adapt to targets of different sizes, and has strong adaptive detection capability and high detection accuracy.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0008] A deep reinforcement learning-based method for particleboard target detection includes:

[0009] Step 1: Obtain the particleboard defect sample image dataset, including the original sample images and the corresponding target defect category labels and target defect location labels; divide the original sample images and target defect category labels into training and testing sets as the target classification dataset for training the classification model, and divide the original sample images and target defect location labels into training and testing sets as the target detection dataset for training the deep reinforcement learning target detection model; the target defect categories include large wood chips, glue spots, sand leakage, scratches, and dust spots;

[0010] Step 2: Define the state space and action space of the reinforcement learning agent; the state space consists of global features, local features, and historical action sequences, which assist the agent in making decisions; the action space includes translation operations, zoom operations, and termination actions, which are applied to the mask of the target location; the mask represents a rectangular bounding box.

[0011] Step 3: Design a dataset loader and build the framework for a deep reinforcement learning object detection model;

[0012] Step 4: Design the target detection evaluation index and reward function;

[0013] Step 5: Train a classification model using the pre-divided target classification dataset to obtain a trained classification model; train a deep reinforcement learning target detection model using the pre-divided target detection dataset, and select the best deep reinforcement learning target detection model as the particleboard target detection model based on the target detection evaluation metrics.

[0014] As a further improvement to the present invention, step 2 specifically includes:

[0015] 2.1 Define the state space of a reinforcement learning agent: The state space consists of a one-dimensional array of three parts, namely global features, local features, and historical action sequences;

[0016] 2.2 Design of Image Feature Extractor: The image feature extractor is based on the ResNet101 network. Its last fully connected layer is removed and used as the backbone network. Pre-trained weight parameters are loaded and the network is set to evaluation mode.

[0017] 2.3 Global Features: The original image is first preprocessed, i.e., scaled to a fixed size, and then normalized using preset mean and standard deviation; the preprocessed image is then processed by the image feature extractor in step 2.2 to output global features;

[0018] 2.4 Local Features: The image content within the Mask first undergoes the scaling and normalization preprocessing operations in step 2.3. The preprocessed image then passes through the image feature extractor in step 2.2 to output local features.

[0019] 2.5 Historical Action Sequence: Each action uses one-hot encoding, that is, in the action space dimension, the position of the executed action is marked as 1, and the other positions are 0; the sequence is vertically stored with the most recent n action codes, the new action code is placed at the top, and the historical codes are shifted down in sequence; the initial historical action sequence is an array of all 1s;

[0020] A i =[0,1,0,0,…,0],H=[A1,A2,…,A 10 (1);

[0021] Among them, A i Here, H is the encoding of the i-th action, and H is the sequence of historical actions.

[0022] 2.6. The global features from step 2.3, the local features from step 2.4, and the historical action sequence from step 2.5 are concatenated along their length. After standardization, the Agent's state S is obtained.

[0023] 2.7 Define the action space of the reinforcement learning agent: The action space includes translation operations, scaling operations, and termination actions; each time the agent takes an action, it samples the action from the action space according to the probability distribution;

[0024] 2.8 Scaling operations include zooming in on all four sides, zooming out on all four sides, zooming in horizontally, zooming out horizontally, zooming in vertically, and zooming out vertically; the width and height changes for each action are α times the width and height of the current Mask (0≤α≤1);

[0025]

[0026] The formulas for scaling around the mask, scaling horizontally, and scaling vertically are shown from left to right. (x1, y1) are the coordinates of the top left corner of the mask, (x2, y2) are the coordinates of the bottom right corner of the mask, (x′1, y′1) are the coordinates of the top left corner of the mask after the update, (x′2, y′2) are the coordinates of the bottom right corner of the mask after the update, and W and H are the width and height of the mask, respectively.

[0027] 2.9 Translation operations include horizontal and vertical movement. Vertical movement includes upward and downward translation, and horizontal movement includes leftward and rightward translation. The width and height changes for each movement are α times the width and height of the current Mask (0≤α≤1).

[0028]

[0029] The formulas for horizontal and vertical movement operations are shown from left to right.

[0030] 2.10. The termination action indicates that the Agent ends the iteration of the current image with the current Mask as the final bounding box.

[0031] As a further improvement to the present invention, step 3 specifically includes:

[0032] 3.1 Design a dataset loader to read sample image data, extract relevant parameters from the target defect category label and target defect location label, and arrange them in a uniform format: [class, x t1 ,x t2 ,y t1 ,y t2 ], class is the category value to which the target in the image belongs, (x t1 ,y t1 (x) represents the coordinates of the top-left corner of the actual location bounding box recorded by the target defect location label. t2 ,y t2 The coordinates of the lower right corner of the actual location frame recorded by the target defect location label;

[0033] 3.2. Build a framework for a deep reinforcement learning target detection model, including an action decision network and a value evaluation network. The action decision network is used to perform action policy selection and baseline value estimation, while the value evaluation network is responsible for evaluating the state value function.

[0034] 3.3 The behavioral decision-making network consists of a backbone network, action heads, and baseline value heads; the backbone network consists of a linear layer, two linear modules, and a root mean square normalization layer.

[0035] 3.4 The linear module consists of a root mean square normalized layer, a linear layer, a quadratic ReLU activation function, a linear layer, and a random deactivation layer;

[0036] ReLUSquared(x) = (ReLU(x)) 2 (4);

[0037] Where ReLUSquared is a quadratic ReLU activation function;

[0038] 3.5. The state S from step 2.6 is processed by the first linear layer of the backbone network in the behavior decision network, outputting feature A1; feature A1 is processed by the first linear module, outputting feature A2; residual connections are merged to obtain A1 + A2 = A3; after processing by the second linear module, feature A4 is output; residual connections are merged to obtain A3 + A4 = A5; and then it passes through the root mean square normalization layer to obtain normalized feature A6.

[0039] 3.6 The action head consists of a linear layer, a quadratic ReLU activation function layer, and another linear layer; the feature A6 from step 3.5 is processed by the action head to obtain feature A7, which is then passed through a SoftMax layer to output the action probability distribution Action. prob ;

[0040] 3.7 The baseline value head consists of a linear layer, a quadratic ReLU activation function layer, and another linear layer; the feature A6 from step 3.5 is processed by the baseline value head to obtain the baseline value output A8;

[0041] 3.8 The value assessment network consists of a backbone network and a state value head; the backbone network consists of a linear layer, six linear modules as described in step 3.4, and a root mean square normalization layer;

[0042] 3.9. In step 2.6, state S is processed by the first linear layer of the backbone network in the value assessment network, outputting feature B1; feature B1 is processed by the first linear module, outputting feature B2; residual connections are merged, B1 + B2 = B3; after processing by the second linear module, feature B4 is output; residual connections are merged, B3 + B4 = B5; after processing by the third linear module, feature B6 is output; residual connections are merged, B5 + B6 = B7; after processing by the fourth linear module, feature B8 is output; residual connections are merged, B7 + B8 = B9; after processing by the fifth linear module, feature B is output. 10 Residual join merging B9+B 10 =B 11 After processing by the sixth linear module, the output feature B is obtained. 12 Residual join merging B 11 +B 12 =B 13 Finally, after passing through the root mean square normalization layer, the normalized feature B is obtained. 14 ;

[0043] 3.10. The state value head is a linear layer; feature B in step 3.9. 14 The state value output B is obtained after processing the state value header. 15 .

[0044] As a further improvement to the present invention, step 4 specifically includes:

[0045] 4.1 The target detection evaluation metrics include multi-class average precision (mAP), recall, and intersection-over-union (IoU).

[0046]

[0047] Where C is the total number of categories, APi The average precision for class i is represented by TP, where i is the class index; TP represents the number of correctly detected targets, and FN represents the number of undetected targets; A and B represent two bounding boxes; Area intersection Area is the intersection area of ​​the two bounding boxes. union Area is the union area of ​​the two bounding boxes; A Area B These are the areas of bounding boxes A and B, respectively;

[0048] 4.2 The reward function is divided into normal action reward and termination action reward, and they are progressively accumulated;

[0049] 4.3 Normal actions include scaling and translation. When these actions are selected, it first checks whether the mask exceeds the original image area; if it does, -r is applied. b The reward;

[0050] 4.4 After a normal action is executed, if the image exceeds the original image range, the Mask will be constrained to the correct range by the constraints.

[0051]

[0052] Where (x1,y1) are the coordinates of the top left corner of the Mask, (x2,y2) are the coordinates of the bottom right corner of the Mask, (x′1,y′1) are the coordinates of the top left corner of the Mask after the update, (x′2,y′2) are the coordinates of the bottom right corner of the Mask after the update, and max(a,b) and min(a,b) are used to get the maximum and minimum values ​​of the array (a,b) respectively.

[0053] 4.5. The IOU between the Mask after normal operation and the actual Mask of the target defect location label is calculated using the formula in step 4.1; the IOU reward function is as follows:

[0054]

[0055] Where i is the index of the i-th action, 1, 2, 3...; β u β is the weight of positive rewards. d The initial IOU is calculated by combining the initial Mask and the real Mask, with the initial Mask size being the original image size.

[0056] 4.6 When a termination action is selected, first set the termination flag done=1, and then give a tiered reward based on the IOU calculated from the previous action;

[0057]

[0058] 4.7 If the Agent selects the terminate action during the first action selection, then grant -r. b The reward.

[0059] As a further improvement to the present invention, step 5 specifically includes:

[0060] 5.1. Train the classification model VGG16 using the target classification dataset from step 1;

[0061] Step 3.1 loads the original image data of the sample and extracts information from the target defect category label and the target defect location label. Steps 2, 3, and 4 build the framework of a complete deep reinforcement learning target detection model and initialize the experience pool, which is used to store past action information.

[0062] 5.2 Select a sample original image, initialize the Mask size to the entire image size, RC Vec Store the coordinates of the top-left and bottom-right corners of the Mask. The historical action sequence H is an n x 10 array of all 1s. The total reward R all The value is assigned to 0, and the termination flag done is also 0;

[0063] 5.3 Calculate the initial IOU0 based on the IOU formula in step 4.1; use the feature extraction method in step 2 to extract global features, local features and historical action sequences respectively, and concatenate them into the initial state S;

[0064] 5.4. State S serves as the input to the behavior decision network and value evaluation network, outputting the action probability distribution and state value v. Then, the agent randomly samples action a based on the action probability distribution and calculates the log probability a of the selected action. log ;

[0065] 5.5 Construct a one-hot vector according to the encoding method in step 2.5, and update the historical action sequence H;

[0066] 5.6. According to steps 2.7 to 2.10, perform action a to update Mask and RC. Vec Then, recalculate the IOU, and then calculate the reward r and the termination flag done according to the reward function in steps 4.2 to 4.7;

[0067] 5.7. Accumulate the single-step reward r and update the total reward R. all ;

[0068] 5.8. Following the feature extraction method in step 2, extract the updated image features as local features, and combine them with the global features of the original image extracted in step 5.3, as well as the updated historical action sequence, to construct the next state S_.

[0069] 5.9、(S,a,r,done,a log (v) is placed into the experience pool as a complete action process;

[0070] 5.10. When the Agent has performed enough actions, past experience, and the next state S... - It will be used to train the Agent, that is, to train a deep reinforcement learning object detection model and clear the experience pool;

[0071] 5.11. If the termination flag done=1 or the maximum number of iterations per image is reached, exit the iterative detection of the current image; after the preprocessing operation in step 2.3, the updated local image is input into the pre-trained classification model VGG16 to identify the target defect category; then select the next image from the training set and jump to step 5.2 to continue the loop; otherwise, proceed to the next state S. - As the current state, jump to step 5.4 and continue the loop;

[0072] 5.12. Once all images in the training set have been traversed, the test set images are traversed using the same method as steps 5.2 to 5.11, skipping the training processes in steps 5.9 and 5.10. After the test set is traversed, mAP and Recall are calculated according to step 4.1.

[0073] 5.13. Steps 5.2 to 5.12 will be iterated Eposide times to select the best detection model parameters based on the object detection evaluation index, thereby obtaining the best deep reinforcement learning object detection model.

[0074] To achieve the above-mentioned technical objectives, another technical solution adopted by the present invention is as follows:

[0075] A deep reinforcement learning-based particleboard target detection system includes:

[0076] The target detection dataset processing module is used to acquire a particleboard defect sample image dataset, which includes the original sample image and the corresponding target defect category label and target defect location label. It is also used to check whether the original image in the particleboard defect sample image dataset matches the target defect category label and target defect location label. Furthermore, it is used to divide the original sample image and the target defect category label into training set and test set as the target classification dataset for training the classification model, and to divide the original sample image and the target defect location label into training set and test set as the target detection dataset for training the deep reinforcement learning target detection model.

[0077] The state and action space definition module is used to construct the state and action space of the reinforcement learning agent;

[0078] The reinforcement learning algorithm building module is used to design a dataset loader and build a framework for deep reinforcement learning object detection models.

[0079] The metrics and rewards module is used to select target detection evaluation metrics and design reward functions.

[0080] The network training module is used to train a classification model using a pre-divided target classification dataset, and within a specified number of iterations, to train a deep reinforcement learning target detection model using a pre-divided target detection dataset, update the network parameters, and select the optimal network parameters based on the target detection evaluation metrics to obtain the optimal deep reinforcement learning target detection model.

[0081] The beneficial effects of this invention are as follows:

[0082] (1) This invention proposes a novel particleboard target detection method and system based on deep reinforcement learning for particleboard target detection. The target detection task is accomplished by continuously transforming the bounding box (rectangular box). Existing single-stage and two-stage detectors increase detection performance at the cost of larger and wider network structures and increased parameter computation. To address these issues, a deep reinforcement learning-based particleboard target detection method is used. Its backbone network consists of only multiple fully connected layers, significantly reducing parameters and computational load. Furthermore, by adjusting the amount of shape transformation and the number of iterations, it can adapt to targets of different sizes and achieve accurate localization.

[0083] (2) This invention optimizes and improves the network architecture of the Agent in reinforcement learning. It uses multiple linear fully connected layers, random deactivation layers, root mean square normalization layers, and a quadratic ReLU activation function to form a linear module. The network structure is clear and efficient. The root mean square normalization layer improves training stability, the quadratic ReLU activation function enhances the network's nonlinear expressive ability, and the random deactivation layer prevents overfitting. Compared to traditional one-stage and two-stage detectors, its overall structure is lighter, computationally efficient, and exhibits good generalization performance.

[0084] (3) This invention proposes a state calculation method with multi-feature fusion, which integrates global features, local features and historical action sequences. Global features are used to guide the Agent to explore the correct position, local features are used to guide the Agent to refine the Mask, and historical action sequences are used to optimize the Agent's action selection process.

[0085] (4) This invention proposes a multi-angle, phased reward function that balances single-step rewards and endpoint rewards. During action, rewards are given to the Agent based on the change of IOU, which helps to locate the target faster; when the target is successfully detected, a larger reward value is given, which helps the Agent to clarify the target task and speed up convergence; when the Mask after the action exceeds the range of the original image, a negative reward value is given, indicating that this action is a dangerous action, thereby gradually avoiding it.

[0086] (5) This invention can be applied to defect (target) detection tasks in particleboard images (defect types include: large wood chips, glue spots, sand leakage, scratches, dust spots), and can accurately detect the defect location. It has strong adaptive detection capability and high detection accuracy. This invention is suitable for single-target detection, that is, when there is one target on a particleboard image. Attached Figure Description

[0087] Figure 1 Flowchart of a particleboard target detection method based on deep reinforcement learning.

[0088] Figure 2 Flowchart for training a particleboard target detection method using deep reinforcement learning

[0089] Figure 3 This is a schematic diagram of the original graph and the region graph in the state space.

[0090] Figure 4 This is a diagram illustrating actions within the action space.

[0091] Figure 5 Network framework diagram for reinforcement learning algorithm.

[0092] Figure 6 This is a schematic diagram of particleboard defect detection using the method of the present invention.

[0093] Figure 7 The images show the detection results for the other four types of defects. Detailed Implementation

[0094] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0095] A deep reinforcement learning-based method for particleboard target detection, such as Figure 1 As shown, it includes:

[0096] Step 1: Obtain the particleboard defect sample image dataset, including the original sample images and the corresponding target defect category labels and target defect location labels; divide the original sample images and target defect category labels into training and testing sets according to a specified ratio, which will be used as the target classification dataset for training the classification model; divide the original sample images and target defect location labels into training and testing sets according to a specified ratio, which will be used as the target detection dataset for training the deep reinforcement learning target detection model; the target defect categories include large wood chips, glue spots, sand leakage, scratches, and dust spots.

[0097] Step 2: Define the state space and action space of the reinforcement learning agent (in reinforcement learning, the agent is a "decision-maker" that learns the optimal behavioral strategy by constantly interacting with the environment); the state space consists of global features, local features, and historical action sequences to assist the agent in making decisions; the action space includes translation operations, zoom operations, and termination actions, which are applied to the mask for locating the target; the mask is a rectangle used to cover the target to be detected.

[0098] Step 3: Design a dataset loader and build the framework for a deep reinforcement learning object detection model.

[0099] Step 4: Based on the task objectives, design the target detection evaluation index and reward function.

[0100] Step 5: Train a classification model using the pre-divided target classification dataset to obtain a trained classification model; train a deep reinforcement learning target detection model using the pre-divided target detection dataset, and select the best deep reinforcement learning target detection model as the particleboard target detection model based on the target detection evaluation metrics.

[0101] Step 1 specifically includes:

[0102] 1.1 Obtain a dataset of particleboard defect sample images, including the original sample images and the target defect category and location labels. Check whether the labels correspond to the images to ensure the accuracy of the candidate label boxes. 1.2 Randomly divide the dataset into training and testing sets according to a specified ratio, ensuring that the image and label pairing for each sample is complete and that there is no data loss or incorrect matching.

[0103] Step 2 specifically includes:

[0104] 2.1 Define the state space of the reinforcement learning agent: The state space consists of a one-dimensional array of three parts, specifically including global features (such as...). Figure 3 Features of the original image shown), local features (such as...) Figure 3 The rectangular box shown contains image features (i.e., region map) and historical action sequences.

[0105] 2.2 Design of Image Feature Extractor: The image feature extractor is based on the ResNet101 network. Its last fully connected layer is removed and used as the backbone network. Pre-trained weight parameters are loaded and the network is set to evaluation mode.

[0106] 2.3 Global Features: The original image is first preprocessed, that is, scaled to a fixed size and then normalized using a preset mean and standard deviation; the preprocessed image is then processed by the image feature extractor in step 2.2 to output global features.

[0107] 2.4 Local Features: The image content within the Mask first undergoes the scaling and normalization preprocessing operations in step 2.3. The preprocessed image then passes through the image feature extractor in step 2.2 to output local features.

[0108] 2.5 Historical Action Sequence: Each action uses one-hot encoding, that is, in the action space dimension, the position of the executed action is marked as 1, and the other positions are 0; the sequence is vertically stored with the most recent n action codes, the new action code is placed at the top, and the historical codes are shifted down in sequence; the initial historical action sequence is an array of all 1s;

[0109] A i =[0,1,0,0,…,0],H=[A1,A2,…,A 10 (1);

[0110] Among them, A i Let H be the encoding of the i-th action, and H be the sequence of historical actions.

[0111] 2.6. The global features from step 2.3, the local features from step 2.4, and the historical action sequence from step 2.5 are concatenated along the length direction. After standardization, the state S of the Agent is obtained.

[0112] 2.7 Define the action space of a reinforcement learning agent: The action space includes, for example, Figure 4 The translation, scaling, and termination actions are shown; each time the Agent takes an action, it samples the action from the action space according to the probability distribution.

[0113] 2.8, such as Figure 4 As shown, the scaling operations include zooming in on all four sides, zooming out on all four sides, zooming in horizontally, zooming out horizontally, zooming in vertically, and zooming out vertically; the width and height changes for each action are α times the width and height of the current Mask (0≤α≤1);

[0114]

[0115] The formulas for scaling around the mask, scaling horizontally, and scaling vertically are shown from left to right. (x1, y1) are the coordinates of the top left corner of the mask, (x2, y2) are the coordinates of the bottom right corner of the mask, (x′1, y′1) are the coordinates of the top left corner of the mask after the update, (x′2, y′2) are the coordinates of the bottom right corner of the mask after the update, and W and H are the width and height of the mask, respectively.

[0116] 2.9, such as Figure 4 As shown, the translation operation includes upward translation, downward translation, leftward translation, and rightward translation; the width and height change of each action is α times the width and height of the current Mask (0≤α≤1);

[0117]

[0118] The formulas for horizontal movement (including left and right translation) and vertical movement (including up and down translation) are listed from left to right; the remaining parameters are the same as in step 2.8.

[0119] 2.10. The termination action indicates that the Agent ends the iteration of the current image with the current Mask as the final bounding box.

[0120] Step 3 specifically includes:

[0121] 3.1 Design a dataset loader to read sample image data, extract relevant parameters from the target defect category label and target defect location label, and arrange them in a uniform format: [class, x t1 ,x t2 ,y t1 ,y t2 ], class is the category value to which the target in the image belongs, (x t1 ,y t1 ) represents the top-left corner coordinates of the actual location bounding box (actual mask) recorded for the target defect location label, (x t2 ,y t2 The coordinates of the lower right corner of the actual location box (actual mask) recorded for the target defect location label are shown.

[0122] 3.2. Construct a deep reinforcement learning framework, namely the framework of a deep reinforcement learning target detection model, which includes an action decision network and a value evaluation network. The action decision network is used to perform action policy selection and baseline value estimation, while the value evaluation network is responsible for evaluating the state value function.

[0123] 3.3, such as Figure 5 As shown, the behavioral decision network consists of a backbone network, an action head, and a baseline value head; the backbone network consists of a linear layer, two linear modules, and a root mean square normalization layer.

[0124] 3.4 The linear module consists of a root mean square normalized layer, a linear layer, a quadratic ReLU activation function, a linear layer, and a random deactivation layer;

[0125] ReLUSquared(x) = (ReLU(x)) 2 (4);

[0126] ReLUSquared is a quadratic ReLU activation function.

[0127] 3.5. The state S in step 2.6 is processed by the first linear layer of the backbone network in the behavior decision network, and outputs feature A1; feature A1 is processed by the first linear module, and outputs feature A2; residual connections are merged A1+A2=A3; after processing by the second linear module, feature A4 is output; residual connections are merged A3+A4=A5; and then after passing through the root mean square normalization layer, normalized feature A6 is obtained.

[0128] 3.6 The action head consists of a linear layer, a quadratic ReLU activation function layer, and another linear layer; the feature A6 from step 3.5 is processed by the action head to obtain feature A7, which is then passed through a SoftMax layer to output the action probability distribution Action. prob .

[0129] 3.7 The baseline value head consists of a linear layer, a quadratic ReLU activation function layer, and a linear layer; the feature A6 from step 3.5 is processed by the baseline value head to obtain the baseline value output A8.

[0130] 3.8, such as Figure 5 As shown, the value assessment network consists of a backbone network and a state value head; the backbone network consists of a linear layer, six linear modules as described in step 3.4, and a root mean square normalization layer.

[0131] 3.9. In step 2.6, state S is processed by the first linear layer of the backbone network in the value assessment network, outputting feature B1; feature B1 is processed by the first linear module, outputting feature B2; residual connections are merged, B1 + B2 = B3; after processing by the second linear module, feature B4 is output; residual connections are merged, B3 + B4 = B5; after processing by the third linear module, feature B6 is output; residual connections are merged, B5 + B6 = B7; after processing by the fourth linear module, feature B8 is output; residual connections are merged, B7 + B8 = B9; after processing by the fifth linear module, feature B is output. 10 Residual join merging B9+B 10 =B 11 After processing by the sixth linear module, the output feature B is obtained. 12 Residual join merging B 11+B 12 =B 13 Finally, after passing through the root mean square normalization layer, the normalized feature B is obtained. 14 .

[0132] 3.10. The state value head is a linear layer; feature B in step 3.9. 14 The state value output B is obtained after processing the state value header. 15 .

[0133] Step 4 specifically includes:

[0134] 4.1 The target detection evaluation metrics include multi-class average precision (mAP), recall, and intersection-over-union (IoU).

[0135]

[0136] Where C is the total number of categories, AP i TP represents the average precision for the i-th class, where i is the class index; FN represents the number of correctly detected targets, and A and B represent two bounding boxes, specifically the actual mask (rectangle) recorded by the target defect location label and the mask (rectangle) calculated by the model; Area intersection Area is the intersection area of ​​the two bounding boxes. union Area is the union area of ​​the two bounding boxes; A Area B These are the areas of bounding boxes A and B, respectively.

[0137] 4.2 The reward function needs to be set according to the task objective and is one of the core components of the reinforcement learning algorithm. The reward function is divided into normal action reward and termination action reward, and they are gradually added together.

[0138] 4.3 Normal actions include scaling and translation. When these actions are selected, it first checks whether the mask exceeds the original image area; if it does, -r is applied. b The reward.

[0139] 4.4 After a normal action is executed, if the image exceeds the original image range, the Mask will be constrained to the correct range by the constraints.

[0140]

[0141] Here, (x1, y1) are the coordinates of the top left corner of the Mask, (x2, y2) are the coordinates of the bottom right corner of the Mask, (x′1, y′1) are the coordinates of the top left corner of the Mask after the update, (x′2, y′2) are the coordinates of the bottom right corner of the Mask after the update, and max(a, b) and min(a, b) are used to get the maximum and minimum values ​​of the array (a, b) respectively.

[0142] 4.5. The IOU between the Mask after normal operation and the actual Mask of the target defect location label is calculated using the formula in step 4.1; the IOU reward function is as follows:

[0143]

[0144] Where i is the index of the i-th action, 1, 2, 3...; β u β is the weight of positive rewards. d The weight for negative rewards is calculated using the initial IOU, which is the initial mask and the real mask. The initial mask size is the original image size.

[0145] 4.6 When a termination action is selected, first set the termination flag done=1, and then give a tiered reward based on the IOU calculated from the previous action;

[0146]

[0147] 4.7 If the Agent selects the terminate action during the first action selection, then grant -r. b The reward.

[0148] like Figure 2 As shown, step 5 specifically includes:

[0149] 5.1. Train the classification model VGG16 using the target classification dataset from step 1. The specific training method for the classification model VGG16 adopts existing techniques.

[0150] Step 3.1 loads the original image data of the sample and extracts information from the target defect category label and the target defect location label. Steps 2, 3, and 4 build the framework of a complete deep reinforcement learning target detection model and initialize the experience pool, which is used to store past action information.

[0151] 5.2 Select a sample original image, initialize the Mask size to the entire image size, RC Vec Store the coordinates of the top-left and bottom-right corners of the Mask. The historical action sequence H is an n x 10 array of all 1s. The total reward R all The value is set to 0, and the termination flag done is also 0.

[0152] 5.3 Calculate the initial IOU0 based on the IOU formula in step 4.1; use the feature extraction method in step 2 to extract global features, local features and historical action sequences respectively, and splice them into the initial state S.

[0153] 5.4. State S serves as the input to the behavior decision network and value evaluation network, outputting the action probability distribution and state value v. Then, the agent randomly samples action a based on the action probability distribution and calculates the log probability a of the selected action. log .

[0154] 5.5 Construct a one-hot vector according to the encoding method in step 2.5, and update the historical action sequence H.

[0155] 5.6. According to steps 2.7 to 2.10, perform action a to update Mask and RC. Vec Then, recalculate the IOU, and then calculate the reward r and the termination flag done according to the reward function in steps 4.2 to 4.7.

[0156] 5.7. Accumulate the single-step reward r and update the total reward R. all The rewards for each action in each image are calculated and accumulated. After switching images, the total reward is reset to 0.

[0157] 5.8 Following the feature extraction method in step 2, extract the features of the updated image (the image within the updated Mask) as local features. Combine these features with the global features extracted from the original image in step 5.3, and the updated historical action sequence to construct the next state S_. Global features refer to the features of the original image, which are extracted only once initially. Local features refer to the features of the updated image.

[0158] 5.9、(S,a,r,done,a log ,v) is placed into the experience pool as a complete action process.

[0159] 5.10. When the Agent has performed enough actions, past experience, and the next state S... - This will be used to train the Agent, i.e., to train a deep reinforcement learning object detection model, and to empty the experience pool. The reward represents the quality of the algorithm's actions; its purpose is to provide the reinforcement learning algorithm with a reference, guiding it to update in the direction of increasing total reward. The reward is used in step 5.10 to train the model. The baseline value output from step 3.7 and the state value output from step 3.10 are used in step 5.10 to calculate the loss, train the model, and update the parameters.

[0160] 5.11. If the termination flag done=1 or the maximum number of iterations per image is reached, the iterative detection of the current image is terminated; the updated local image (the image within the updated Mask) is preprocessed in step 2.3 and then input into the pre-trained classification model VGG16 to identify the target defect category.

[0161] Then select the next image from the training set, jump to step 5.2, and continue the loop. Otherwise, proceed to the next state S. - As the current state, jump to step 5.4 and continue the loop.

[0162] 5.12. Once all images in the training set have been traversed, repeat steps 5.2-5.8 and 5.11 to traverse the images in the test set. After the test set has been traversed, calculate mAP and Recall according to step 4.1.

[0163] 5.13. Steps 5.2 to 5.12 will be repeated Eposide times to select the best detection model parameters (parameters in the best behavior decision network and value evaluation network) based on the object detection evaluation metrics (mAP and Recall) to obtain the best deep reinforcement learning object detection model.

[0164] In actual prediction, the original image to be tested is processed in the same way as steps 5.2-5.8 (during which state S is input into the behavior decision network and value evaluation network of the best deep reinforcement learning target detection model) and step 5.11, and the final updated local image is output, which is recorded as the final detected target defect. After the preprocessing operation in step 2.3, the updated local image is input into the pre-trained classification model VGG16 to identify the target defect category.

[0165] Figure 6 This is a schematic diagram illustrating the process of detecting target defects (glue spots) using the method of this invention on an original particleboard image. Figure 7 This is a schematic diagram illustrating the detection results for four other types of defects (the target defects in the images within the red mask from left to right are, in order, dust spots, sand leakage, scratches, and large wood shavings). Figures 6-7 As can be seen, the method of the present invention can be applied to defect (target) detection tasks in particleboard images (defect types include: large wood chips, glue spots, sand leakage, scratches, and dust spots), accurately detects the defect location, has strong adaptive detection capability, and high detection accuracy. The present invention is suitable for single-target detection, that is, when there is only one target on a particleboard image.

[0166] This embodiment also provides a deep reinforcement learning-based particleboard target detection system, including:

[0167] The target detection dataset processing module is used to acquire a particleboard defect sample image dataset, which includes the original sample image and the corresponding target defect category label and target defect location label. It is also used to check whether the original image in the particleboard defect sample image dataset matches the target defect category label and target defect location label. Furthermore, it is used to divide the original sample image and the target defect category label into training set and test set as the target classification dataset for training the classification model, and to divide the original sample image and the target defect location label into training set and test set as the target detection dataset for training the deep reinforcement learning target detection model.

[0168] The state and action space definition module is used to construct the state and action space of the reinforcement learning agent;

[0169] The reinforcement learning algorithm building module is used to design a dataset loader and build a framework for deep reinforcement learning object detection models.

[0170] The metrics and rewards module is used to select target detection evaluation metrics and design reward functions.

[0171] The network training module is used to train a classification model using a pre-divided target classification dataset, and within a specified number of iterations, to train a deep reinforcement learning target detection model using a pre-divided target detection dataset, update the network parameters, and select the optimal network parameters based on the target detection evaluation metrics to obtain the optimal deep reinforcement learning target detection model.

[0172] The scope of protection of this invention includes, but is not limited to, the above embodiments. The scope of protection of this invention is defined by the claims. Any substitutions, modifications, or improvements to this technology that are easily conceived by those skilled in the art fall within the scope of protection of this invention.

Claims

1. A deep reinforcement learning-based method for particleboard target detection, characterized in that, include: Step 1: Obtain the particleboard defect sample image dataset, including the original sample images and the corresponding target defect category labels and target defect location labels; The original sample images and target defect category labels are divided into training and testing sets, which are used as target classification datasets for training classification models. The original sample images and target defect location labels are divided into training and testing sets, which are used as target detection datasets for training deep reinforcement learning target detection models. Step 2: Define the state space and action space of the reinforcement learning agent; the state space consists of global features, local features, and historical action sequences to assist the agent in making decisions; the action space includes translation operations, zoom operations, and termination actions, which are applied to the mask of the target location. Step 3: Design a dataset loader and build the framework for a deep reinforcement learning object detection model; specifically including: 3.1 Design a dataset loader to read sample image data, extract relevant parameters from the target defect category label and target defect location label, and arrange them in a uniform format: , Assign a value to the category to which the target in the image belongs. The coordinates of the top left corner of the actual location box recorded for the target defect location label. The coordinates of the lower right corner of the actual location frame recorded for the target defect location label; 3.

2. Build a framework for a deep reinforcement learning target detection model, including an action decision network and a value evaluation network. The action decision network is used to perform action policy selection and baseline value estimation, while the value evaluation network is responsible for evaluating the state value function. 3.3 The behavioral decision network consists of a backbone network, action heads, and baseline value heads; the backbone network of the behavioral decision network consists of a linear layer, two linear modules, and a root mean square normalization layer. 3.4 The linear module consists of a root mean square normalized layer, a linear layer, a quadratic ReLU activation function, a linear layer, and a random deactivation layer connected in sequence. （4）； Where ReLUSquared is a quadratic ReLU activation function; 3.5 Agent Status After processing by the first linear layer of the backbone network in the behavioral decision network, the output features are... ;feature After processing by the first linear module, the output features are... Residual join merging After processing by the second linear module, the output features are... Residual join merging After passing through the root mean square normalization layer, normalized features are obtained. ; 3.6 The action head consists of a linear layer, a quadratic ReLU activation function layer, and another linear layer connected in sequence; features from step 3.

5. Features are obtained after motion head processing. The action probability distribution is then output through the SoftMax layer. ; 3.7 The baseline value header consists of a linear layer, a quadratic ReLU activation function layer, and another linear layer connected in sequence; features from step 3.

5. The baseline value output is obtained after baseline value head processing. ; 3.8 The value assessment network consists of a backbone network and a state value head; the backbone network of the value assessment network consists of a linear layer, six linear modules as described in step 3.4, and a root mean square normalization layer; 3.9 Agent Status After processing by the first linear layer of the backbone network in the value assessment network, the output features are... ;feature After processing by the first linear module, the output features are... Residual join merging After processing by the second linear module, the output features are... Residual join merging After processing by the third linear module, the output features are... Residual join merging After processing by the fourth linear module, the output features are... Residual join merging After processing by the fifth linear module, the output features are... Residual join merging After processing by the sixth linear module, the output features are... Residual join merging Finally, after passing through the root mean square normalization layer, the normalized features are obtained. ; 3.

10. The state value head is a linear layer; features in step 3.

9. The state value output is obtained after processing the state value head. ; Step 4: Design the target detection evaluation index and reward function; Step 5: Train a classification model using the divided target classification dataset to obtain a trained classification model; train a deep reinforcement learning target detection model using the divided target detection dataset, and select the best deep reinforcement learning target detection model as the particleboard target detection model based on the target detection evaluation metrics.

2. The deep reinforcement learning-based particleboard target detection method according to claim 1, characterized in that, Step 2 specifically includes: 2.1 Define the state space of a reinforcement learning agent: The state space consists of a one-dimensional array of three parts, namely global features, local features, and historical action sequences; 2.2 Design of Image Feature Extractor: The image feature extractor is based on the ResNet101 network. Its last fully connected layer is removed and used as the backbone network. Pre-trained weight parameters are loaded and the network is set to evaluation mode. 2.3 Global Features: The original image is first preprocessed, i.e., scaled to a fixed size, and then normalized using preset mean and standard deviation; the preprocessed image is then processed by the image feature extractor in step 2.2 to output global features; 2.4 Local Features: The image content within the Mask first undergoes the scaling and normalization preprocessing operations in step 2.

3. The preprocessed image then passes through the image feature extractor in step 2.2 to output local features. 2.5 Historical Action Sequence: Each action uses one-hot encoding, meaning that in the action space dimension, the position of the executed action is marked as 1, and the remaining positions are marked as 0; the sequence is vertically stored with the nearest... Each action is encoded, with the new action code placed at the top and the historical codes shifted down sequentially; the initial historical action sequence is an array of all 1s. （1）； in, For the first Encoding of the next action. ; 2.

6. The global features from step 2.3, the local features from step 2.4, and the historical action sequence from step 2.5 are concatenated along their length. After standardization, the Agent's state is obtained. ; 2.7 Define the action space of the reinforcement learning agent: The action space includes translation operations, scaling operations, and termination actions; each time the agent takes an action, it samples the action from the action space according to the probability distribution; 2.8 Scaling operations include zooming in on all four sides, zooming out on all four sides, zooming in horizontally, zooming out horizontally, zooming in vertically, and zooming out vertically; the width and height changes for each action are equal to the current width and height of the Mask. times, ; （2）； The formulas for scaling around the perimeter, horizontally, and vertically, from left to right, are shown below. The coordinates of the top left corner of the Mask. The coordinates of the bottom right corner of the Mask. The coordinates of the top left corner after the Mask is updated. The coordinates of the bottom right corner after the Mask is updated. These are the width and height of the Mask, respectively; 2.9 Translation operations include horizontal and vertical movement. Vertical movement includes upward and downward translation, and horizontal movement includes leftward and rightward translation. The width and height changes for each movement are equal to the current width and height of the Mask. times, ; （3）； The formulas for horizontal and vertical movement operations are shown from left to right. 2.

10. The termination action indicates that the Agent ends the iteration of the current image with the current Mask as the final bounding box.

3. The deep reinforcement learning-based particleboard target detection method according to claim 2, characterized in that, Step 4 specifically includes: 4.1 The target detection evaluation index adopts the average accuracy of multiple categories. Recall rate and intersection ; （5）； （6）； （7）； in, The total number of categories, Indicates the first The average precision of the class It is a category index; This indicates the number of targets that were correctly detected. That is, the number of undetected targets; , Indicates two bounding boxes; The area of ​​the intersection of the two bounding boxes; The area of ​​the union of the two bounding boxes; , These are the bounding boxes. and The area; 4.2 The reward function is divided into normal action reward and termination action reward, and they are progressively accumulated; 4.3 Normal actions include scaling and translation. When these actions are selected, the system first checks if the mask exceeds the original image area; if it does, it applies an error. The reward; 4.4 After a normal action is executed, if the image exceeds the original image range, the Mask will be constrained to the correct range by the constraints. （8）； in, The coordinates of the top left corner of the Mask. The coordinates of the bottom right corner of the Mask. The coordinates of the top left corner after the Mask is updated. The coordinates of the bottom right corner after the Mask is updated. , Used to retrieve arrays respectively The maximum and minimum values; 4.5 The Mask after normal operation and the actual Mask of the target defect location label are calculated using the formula in step 4.

1. ; The reward function is as follows: （9）； in, For the first Index of the next action. ; As the weight of positive rewards, The weight of negative rewards, initially Calculated from the initial mask and the real mask, the initial mask size is the original image size; 4.6 When a termination action is selected, first set the termination flag. Calculated based on the previous action Provide tiered rewards; （10）； 4.7 If the Agent selects the terminate action during the first action selection, then give... The reward.

4. The deep reinforcement learning-based particleboard target detection method according to claim 3, characterized in that, Step 5 specifically includes: 5.

1. Train the classification model VGG16 using the target classification dataset from step 1; Step 3.1 loads the original image data of the sample and extracts information from the target defect category label and the target defect location label. Steps 2, 3, and 4 build the framework of the complete deep reinforcement learning target detection model and initialize the experience pool, which is used to store past action information. 5.2 Select a sample original image and initialize the Mask size to the entire image size. Store the coordinates of the top-left and bottom-right corners of the Mask, and the total reward. Assigning a value of 0 as the termination flag. It is also 0; 5.3 Based on step 4.1 Formula calculation initial The feature extraction method from step 2 is used to extract global features, local features, and historical action sequences, which are then concatenated to form the initial state. ; 5.4 Status As input to the behavioral decision-making network and the value assessment network, the outputs are the action probability distribution and state value. Then the agent randomly samples actions based on the action probability distribution. And calculate the log probability of the selected action. ; 5.5 Construct one-hot vectors according to the encoding method in step 2.5, and update the historical action sequence. ; 5.

6. Perform the actions according to steps 2.7 to 2.

10. Update Mask and and recalculate Then, based on the reward function from steps 4.2 to 4.7, calculate the reward. With termination mark ; 5.7 Cumulative Single-Step Rewards Update total rewards ; 5.

8. Following the feature extraction method in step 2, extract the updated image features as local features, and combine them with the global features of the original image extracted in step 5.3, as well as the updated historical action sequence, to construct the next state S_. 5.9 It was placed into the experience pool as a complete action process; 5.

10. When the agent has performed enough actions, past experience, and the next state... It will be used to train the Agent, that is, to train a deep reinforcement learning object detection model and clear the experience pool; 5.

11. If the termination flag is present If the maximum number of iterations per image is reached, the iterative detection of the current image is terminated; the updated local image, after the preprocessing operation in step 2.3, is input into the pre-trained classification model VGG16 to identify the target defect category; then, the next image is selected from the training set, and the process jumps to step 5.2 to continue the loop; otherwise, the process will proceed to the next state. As the current state, jump to step 5.4 and continue the loop; 5.

12. Once all images in the training set have been traversed, repeat steps 5.2-5.8 and 5.11 to traverse the images in the test set. After traversing all images in the test set, calculate according to step 4.

1. and ; 5.

13. Steps 5.2 to 5.12 will be repeated. Next, the optimal detection model parameters are selected based on the target detection evaluation index to obtain the optimal deep reinforcement learning target detection model.

5. A particleboard target detection system for implementing the deep reinforcement learning method for particleboard target detection as described in claim 1, characterized in that, include: The target detection dataset processing module is used to acquire a particleboard defect sample image dataset, which includes the original sample image and the corresponding target defect category label and target defect location label. It is also used to check whether the original image in the particleboard defect sample image dataset matches the target defect category label and target defect location label. Furthermore, it is used to divide the original sample image and the target defect category label into a training set and a test set, which are used as the target classification dataset for training the classification model, and to divide the original sample image and the target defect location label into a training set and a test set, which are used as the target detection dataset for training the deep reinforcement learning target detection model. The state and action space definition module is used to construct the state and action space of the reinforcement learning agent; The reinforcement learning algorithm building module is used to design a dataset loader and build a framework for deep reinforcement learning object detection models. The metrics and rewards module is used to select target detection evaluation metrics and design reward functions. The network training module is used to train a classification model using a pre-divided target classification dataset, and within a specified number of iterations, to train a deep reinforcement learning target detection model using a pre-divided target detection dataset, update the network parameters, and select the optimal network parameters based on the target detection evaluation metrics to obtain the optimal deep reinforcement learning target detection model.

Images