Switch panel target recognition and one-meter line segmentation method based on multi-task learning

Abstract

The method for switch panel target identification and one-meter line segmentation based on multi-task learning belongs to the field of artificial intelligence and building engineering technology, and comprises the following steps: step 1, establishing a switch panel and building one-meter line pixel-level data set; step 2, augmenting the image data set to enhance the generalization ability of the network; step 3, constructing a lightweight multi-task deep learning model combining building switch panel target identification and building one-meter line segmentation; step 4, training the designed deep learning model by using the augmented image data set in step 2; and step 5, performing target identification and image segmentation by using the trained model and outputting the results. The deep learning model with high recognition rate, high segmentation accuracy, low network parameters and real-time performance can effectively improve the recognition accuracy of the switch panel and the accuracy of the one-meter line segmentation, and not only realizes sharing of a main network to complete multi-task, but also reduces the parameter quantity.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and architectural engineering technology, specifically relating to a method for target recognition and one-meter line segmentation of switch panels based on multi-task learning. Background Technology

[0002] With the rapid rise of new technologies such as object-oriented programming, neural networks, and deep learning, scholars are combining computer technology, image processing technology, and network communication technology for application in the construction field. Deep learning, in particular, allows neural network architectures to learn representations of multi-level abstract data, significantly improving technologies such as visual target recognition. Li Mengfeng et al. proposed a method for precise positioning and recognition of substation disconnect switches based on an improved random forest algorithm, achieving a 9% improvement in recognition accuracy, reaching 99.5%. Liu Jiawei et al. proposed an improved fully convolutional network method for wire recognition. This method can extract the optical image features of wires and, compared with traditional machine learning methods, can accurately extract wires from the scene, making the recognition results more reliable. Furthermore, the final model is compressed to improve recognition efficiency, providing theoretical guidance and technical support for straight line recognition in real-world applications.

[0003] In domestic research on building recognition, the research focus is mainly on the establishment of three-dimensional space, and there is no systematic research on specific small objects on two-dimensional walls. Furthermore, due to the lack of pixel-level wall data, the recognition accuracy, segmentation accuracy, and generalization ability of recognition and segmentation models built on small-scale specific wall data sets need to be improved. Regarding the issues of multi-task and lightweight design, existing research uses multiple models to train and complete each branch task separately, which has the problems of large number of parameters and difficulty in porting to embedded devices. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention discloses a method for target recognition and one-meter line segmentation of switch panels based on multi-task learning. This method solves the problems mentioned in the background section by establishing a building switch recognition and one-meter line segmentation model with high recognition rate, high segmentation accuracy, low network parameters, and real-time performance.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] The method for target recognition and one-meter line segmentation of switch panels based on multi-task learning includes the following steps:

[0007] Step 1: Create a pixel-level dataset of switch panels and building one-meter lines;

[0008] Step 2: Expand the image dataset to enhance the network's generalization ability;

[0009] Step 3: Construct a lightweight multi-task deep learning model that combines target recognition of building switch panels with segmentation of the one-meter line of buildings;

[0010] Step 4: Train the designed deep learning model using the expanded image dataset from Step 2;

[0011] Step 5: Use the trained model to perform target recognition and image segmentation, and output the results.

[0012] Preferably, step 2 is a data augmentation method based on geometric transformation and image manipulation, and uses random probability and appropriate combination strategies to perform offline data augmentation on the dataset. At the same time, it uses graphics technology to create simulated images to increase the amount of data, and uses persistence of vision raytracer technology to render a large number of simulated images to maintain generalization ability.

[0013] Preferably, in step 2, the method of data augmentation using simulated images is as follows: using 2000 visible light images without the one-meter line and switch from the real dataset as the background, image classification labels are generated based on the position ratio of the switch on the wall, and corresponding pixel classification labels are generated. During the simulation, 16x oversampling anti-aliasing is used to ensure the similarity between the simulated switch panel and the one-meter line and the real switch panel and the one-meter line, and to increase the jaggedness of the simulated switch panel and reduce the jaggedness of the one-meter line.

[0014] Preferably, in step 2, the specific method for data augmentation using simulated images includes: creating a buffer with a side length of 16 times; randomly adding 0 to 6 switch panels with a side length of 16 times and 0 to 1 one-meter line with a side length of 16 times to this buffer; after Gaussian blurring of the buffer, selecting 256 pixels in each 16×16 region and mixing them into one pixel, so that the color transition of the switch panels and the one-meter line in the entire simulated image tends to be smooth; after adjusting the brightness and obtaining the corresponding labels, superimposing it on the selected background image to generate an enhanced image.

[0015] Preferably, in step 3, when designing a lightweight multi-task deep learning model combining target recognition of building switch panels and segmentation of building one-meter lines, the principle of lightweight design is followed. An improved Yolov5s network is used to globally share a single encoder, which consists of a Backbone network and a Neck network. At the input end, Mosaic data augmentation and adaptive anchor box calculation are employed. The Backbone network uses a Focus structure and a CSP (Cross Stage Partial) structure, while the Neck network uses an FPN+SPPF (Feature Pyramid Networks+Spatial Pyramid Pooling-Fast) structure. The Prediction loss function is calculated using GIOU_Loss. The decoder is divided into two branches: a target detection branch and a segmentation branch. The target detection branch uses CIOU_Loss as the bounding box loss function. In the post-processing of target detection, NMS (Non Maximum Segmentation Loss) is used to filter many target boxes. The Suppression operation selects the most suitable target bounding box; the segmentation branch uses the same network structure as the detection head; the bottom layer of the FPN is fed to the segmentation branch, with a size of (W / 8, H / 8, 256); after three upsampling operations, a feature map with a size of (W, H, 2) is output. Simultaneously, the upsampling layer uses nearest-neighbor interpolation to reduce computational cost. This achieves high-precision output while maintaining high inference speed.

[0016] Preferably, in step 3, a parameter hard-sharing mechanism is adopted. The deep learning model consists of an encoder structure and a decoder structure. Subtasks share the encoder structure parameters. The decoder includes a building switch panel recognition subtask and a one-meter line segmentation subtask. Each subtask has its own parameters. The encoder network is used to extract features of the building switch panel position information and the one-meter line edge information. The decoder network outputs the building switch panel position information and the one-meter line edge information.

[0017] Preferably, in step 4, the amplified image dataset from step 2 is used to train the designed deep learning model. The deep learning model has a maximum iteration count of 100,000 steps, uses RMSProp (Root Mean Square Propagation), and has a learning rate of 0.01 that decays exponentially.

[0018] The beneficial effects of this invention's method for target recognition and one-meter line segmentation of switch panels based on multi-task learning are as follows: By establishing a deep learning model, this invention greatly reduces the number of parameters, improves the recognition speed, achieves simultaneous processing of two tasks, and provides a technical method for subsequent porting to embedded devices. Addressing the issues of multi-task and lightweight design, unlike existing research that uses multiple models to train and complete each branch task separately, this invention achieves fewer parameters through a hard parameter sharing mechanism. It not only achieves multi-task completion by sharing a single backbone network but also reduces the number of parameters, making it portable to embedded devices. During data augmentation and fabrication, simulation data technology is used to enhance the dataset from multiple perspectives, improving the model's generalization ability. The model's accuracy reaches 98.5%, precision reaches 99.0%, recall reaches 98.6%, and F1-score reaches 98.7%, outperforming existing research in both recognition accuracy and model size. Attached Figure Description

[0019] Figure 1 Example of a dataset image.

[0020] Figure 2 A diagram illustrating the overall process of data augmentation.

[0021] Figure 3 Image enhancement example.

[0022] Figure 4 Yolov5s model structure diagram;

[0023] Figure 5 FPN model structure diagram;

[0024] Figure 6 The multi-task deep learning model structure of this invention;

[0025] Figure 4 English explanation

[0026] CBS stands for Conv+BatchNorm+SiLU, where Conv refers to convolution, BatchNorm refers to batch normalization, and SiLU refers to activation function.

[0027] Bottleneck Convolutional Programming (CSP) splits the original input into two branches, performs convolution operations on each branch to halve the number of channels, and then performs a Bottleneck*N operation on one branch. Finally, the two branches are concatenated so that the input and output of BottleneckCSP are of the same size. This is to allow the model to learn more features. There are two designs for CSP: CSP1_X and CSP2_X.

[0028] Resunit consists of x residual components.

[0029] SPPF, where SPP stands for Spatial Pyramid Pooling, can transform feature maps of arbitrary size into feature vectors of fixed size. SPPF is also much faster than SPP.

[0030] Maxpool is the maximum pooling layer. The most intuitive benefits of pooling layers are dimensionality reduction, parameter reduction, removal of redundant information, feature compression, simplification of network complexity, reduction of computation, and reduction of memory consumption, among other things.

[0031] Figure 5 English explanation

[0032] Predict represents the predicted output of each layer. Conv represents the convolutional layer.

[0033] Figure 6 English explanation

[0034] Backbone refers to the backbone network, which is part of the network. This backbone network is responsible for extracting information from images for use by subsequent networks.

[0035] The Neck is located between the backbone and the head, and it is designed to better utilize the features extracted from the backbone.

[0036] Building a meter line segment head refers to a meter line segmentation detection head used in construction.

[0037] Wall switch detect head refers to a device that detects switches on walls. Detailed Implementation

[0038] The following description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0039] In the initial embodiment, the method for target recognition and one-meter line segmentation of switch panels based on multi-task learning includes the following steps:

[0040] Step 1: Create a pixel-level dataset of switch panels and building one-meter lines;

[0041] Step 2: Expand the image dataset to enhance the network's generalization ability;

[0042] Step 3: A lightweight multi-task deep learning model combining target recognition of building switch panels and segmentation of building one-meter lines was constructed.

[0043] Step 4: Train the designed deep learning model using the expanded image dataset from Step 2;

[0044] Step 5: Use the trained model to perform target recognition and image segmentation, and output the results.

[0045] In a further embodiment, in step 1, as follows: Figure 1 As shown: Regarding the identification of building switches and the segmentation of the building's one-meter line in China, the building's one-meter line is calculated by adding one meter to the building's surface (not by measuring upwards from the structural surface, but by using elevation lines to control each surface). For example, if the ground elevation of the second floor is 5.9m and the top elevation of the second floor's structure is 5.85m, then the building's one-meter line for the second floor is 6.9m. This is because these elevations should be directly measured from a reference elevation, not measured upwards by 1.05m from the ground. The building's one-meter line is also an important standard line for interior decoration and exterior installation. Therefore, its sample is simple, and the simulation images are highly feasible. The sample collection location is in Qingdao, China. The method involves collecting images of existing building switch panels and the wall surface of the building's one-meter line, creating 800 images, and then creating 400 images using simulation technology.

[0046] In a further embodiment, step 2 is a data augmentation method based on geometric transformation and image manipulation, and uses random probability and appropriate combination strategies to perform offline data augmentation on the dataset. At the same time, it uses graphics technology to create simulated images to increase the amount of data, and uses persistence of vision raytracer (using ray tracing to draw 3D images) technology to render a large number of simulated images to maintain generalization ability.

[0047] The specific process of data augmentation based on Mosaic data augmentation: As the training dataset expands, the performance of deep convolutional neural networks will be further improved. Yolov5s uses Mosaic data augmentation. In this invention, to ensure that a small dataset meets the needs of model training, Mosaic data augmentation uses four images, which are randomly scaled, cropped, and arranged to create a stitched image. The augmentation effect is as follows: Figure 3 As shown.

[0048] In a further embodiment, in step 2, the method of data augmentation using simulated images is as follows: using 2000 visible light images without the one-meter line and switch from the real dataset as the background, image classification labels are generated based on the position ratio of the switch on the wall, and corresponding pixel classification labels are generated. During the simulation, 16x oversampling anti-aliasing is used to ensure the similarity between the simulated switch panel and the one-meter line and the real switch panel and the one-meter line, and to increase the jaggedness of the simulated switch panel and reduce the jaggedness of the one-meter line.

[0049] In a further embodiment, the specific method for data augmentation using simulated images in step 2 includes: creating a buffer with a side length of 16 times; randomly adding 0 to 6 switch panels with a side length of 16 times and 0 to 1 one-meter line with a side length of 16 times to this buffer; after Gaussian blurring of the buffer, selecting 256 pixels in each 16×16 region and mixing them into one pixel, so that the color transition of the switch panels and the one-meter line in the entire simulated image tends to be smooth; after adjusting the brightness and obtaining the corresponding labels, superimposing it on the selected background image to generate an image.

[0050] In a further embodiment, when designing a lightweight multi-task deep learning model combining building switch panel target recognition and building one-meter line segmentation in step 3, the lightweight principle is followed. An improved Yolov5s network is used to globally share a single encoder, which consists of a Backbone network and a Neck network. At the input end, Mosaic data augmentation and adaptive anchor box calculation are used. The Backbone network adopts a Focus structure and a CSP structure, and the Neck network adopts an FPN+SPPF structure. The Prediction loss function is calculated using GIOU_Loss. The decoder is divided into two branches: a target detection branch and a segmentation branch. The target detection branch uses CIOU_Loss for bounding. The loss function for the bounding box is used in the post-processing of object detection. For filtering many bounding boxes, NMS (Non-Maximum Suppression) is performed to select the most suitable bounding box. In the segmentation branch, the image segmentation head uses the same network structure as the detection head. The bottom layer of the FPN is fed to the segmentation branch, with a size of (W / 8, H / 8, 256). After three upsampling operations, a feature map with a size of (W, H, 2) is output. Simultaneously, the upsampling layer uses nearest-neighbor interpolation to reduce computational cost. This achieves high-precision output while maintaining high inference speed.

[0051] In a further embodiment, in step 3, one encoder and two decoders are used as a parameter hard-sharing mechanism for multi-task learning in deep learning. Multi-task learning utilizes the correlation between sub-tasks to improve the performance of each sub-task. Multi-task learning in deep learning is divided into parameter soft-sharing mechanism and parameter hard-sharing mechanism. In the soft-sharing mechanism, each task has its own model and parameters, while in the hard-sharing mechanism, the parameters of the encoder backbone network are shared. Each task's output layer has different parameters. Therefore, the parameter hard-sharing mechanism has fewer parameters. The network designed in this invention adopts the parameter hard-sharing mechanism. The deep learning model proposed in this invention consists of an encoder structure and a decoder structure. Sub-tasks share the encoder structure parameters. The decoder includes a building switch panel recognition sub-task and a one-meter line segmentation sub-task. Each sub-task has its own parameters. The encoder network is used to extract features from the building switch panel position information and the one-meter line edge information, and the decoder network outputs the building switch panel position information and the one-meter line edge information.

[0052] In a further embodiment, in step 3, to meet the requirements of a lightweight deep model, the YOLOv5s backbone is used. YOLOv5s is a single-stage object detection algorithm. Input: During model training, Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling are used. Baseline network: A fusion of Focus and CSP structures. Neck network: Both the object detection and semantic segmentation Neck networks use SPPF+FPN structures. Head output layer: The anchor box mechanism used in the object detection network output layer is the same as YOLOv4, with the addition of a PAN structure. The semantic segmentation layer directly inherits the FPN output from the Neck and extracts semantic information through 3x upsampling using bilinear interpolation.

[0053] In a further embodiment, in step 3, Yolov5s is used to extract the location information of building switch panels. For the extraction of the one-meter line on the building, this invention uses the Yolov5s Backbone as the encoder, and the output feature layer is upsampled three times to complete the extraction of the one-meter line on the building. The multi-task deep learning model structure is as follows: Figure 4 As shown.

[0054] In a further embodiment, the specific design for training the image dataset on the designed network model is as follows: For the multi-task deep learning model, different training methods are adopted for the two different tasks. For image detection, many training strategies used in the Yolov5s source code are employed, as shown in Table 1.

[0055] Table 1 Target Detection Training Strategy

[0056]

[0057]

[0058] For semantic segmentation model training, the FPN training method is adopted. The model trained with Yolov5s is used as the initialization. On this basis, the image size is tripled by upsampling (from 20*20*1024 to 60*60*1024). Then, it is passed through a 1*1*2048 convolutional layer to convert 1024 channels into 2048 channels. The training process is shown in Table 2.

[0059] Table 2. Semantic segmentation model training

[0060]

[0061] The hardware and software used in this invention are shown in Table 3.

[0062] Table 3. Operating Environment Related Parameters

[0063]

[0064] The backbone network parameters of the YOLOv5s model consist of a Focus structure and a CSP structure. The Focus structure is composed of ConvBNSiLN(Conv+BatchNorm+SiLU), and the CSP structure is composed of (ConvBNSiLN+BottleNeck+Concat), where BottleNeck is composed of (ConvBNSiLN+ConvBNSiLN+Concat). The YOLOv5s backbone model is shown in Table 4.

[0065] Table 4. Yolov5s Backbone Structural Model

[0066]

[0067]

[0068] In a further embodiment, in step 4, the amplified image dataset from step 2 is used to train the designed deep learning model. The deep learning model has a maximum iteration count of 100,000 steps, uses RMSProp (Root Mean Square Propagation), and has a learning rate of 0.01 that decays exponentially.

Claims

1. A method for switchboard target recognition and one-meter line segmentation based on multi-task learning, characterized in that: Includes the following steps: Step 1: Create a pixel-level dataset of switch panels and building one-meter lines; Step 2: Expand the image dataset to enhance the network's generalization ability; Step 3: Construct a lightweight multi-task deep learning model that combines target recognition of building switch panels with segmentation of the one-meter line of buildings; Step 4: Train the designed deep learning model using the expanded image dataset from Step 2; Step 5: Use the trained model to perform target recognition and image segmentation, and output the results; Step 2 is a data augmentation method based on geometric transformation and image manipulation. It uses random probability and appropriate combination strategies to perform offline data augmentation on the dataset. At the same time, it uses graphics technology to create simulated images to increase the amount of data and uses persistence of vision raytracer technology to render a large number of simulated images to maintain generalization ability. In step 2, the method of data augmentation using simulated images is as follows: 2000 visible light images without the one-meter line and switch from the real dataset are used as the background. Based on this, image classification labels are generated according to the position ratio of the switch on the wall, and corresponding pixel classification labels are generated. During the simulation, 16x oversampling anti-aliasing is used to ensure the similarity between the simulated switch panel and the one-meter line and the real switch panel and the one-meter line, and to increase the jaggedness of the simulated switch panel and reduce the jaggedness of the one-meter line. In step 3, a parameter hard-sharing mechanism is adopted. The deep learning model consists of an encoder structure and a decoder structure. Subtasks share the encoder structure parameters. The decoder includes a building switch panel recognition subtask and a one-meter line segmentation subtask. Each subtask has its own parameters. The encoder network is used to extract features of the building switch panel position information and the one-meter line edge information. The decoder network outputs the building switch panel position information and the one-meter line edge information.

2. The method for target recognition and one-meter line segmentation of switch panels based on multi-task learning as described in claim 1, characterized in that: In step 2, the specific method for data augmentation using simulated images includes: creating a buffer with a side length of 16 times; randomly adding 0 to 6 switch panels with a side length of 16 times and 0 to 1 one-meter line with a side length of 16 times to this buffer; after Gaussian blurring of the buffer, selecting 256 pixels in each 16×16 region and mixing them into one pixel, so that the color transition of the switch panels and the one-meter line in the entire simulated image tends to be smooth; after adjusting the brightness and obtaining the corresponding labels, superimposing it on the selected background image to generate an enhanced image.

3. The method for target recognition and one-meter line segmentation of switch panels based on multi-task learning as described in claim 2, characterized in that: When designing a lightweight multi-task deep learning model that combines building switch panel target recognition and building one-meter line segmentation in step 3, the lightweight principle is followed, and an improved Yolov5s network is used to share a single encoder globally. This encoder consists of a Backbone network and a Neck network. At the input end, adaptive anchor box calculation is used after data augmentation. The Backbone network adopts Focus and CSP structures, the Neck network adopts FPN+SPPF structure, and the Prediction loss function is calculated using GIOU_Loss. The decoder is divided into two branches: an object detection branch and a segmentation branch. The object detection branch uses CIOU_Loss as the loss function for the bounding box. In the post-processing of object detection, NMS operation is performed to select the most suitable target bounding box from many target boxes. The bottom layer of FPN is fed to the segmentation branch, with a size of (W / 8, H / 8, 256). After three upsampling steps, the output feature map is given, with a size of (W, H, 2). At the same time, the upsampling layer uses nearest-neighbor interpolation to reduce computational cost, achieving high-precision output while maintaining high inference speed.

4. The method for target recognition and one-meter line segmentation of switch panels based on multi-task learning as described in claim 3, characterized in that: In step 4, the amplified image dataset from step 2 is used to train the designed deep learning model. The deep learning model has a maximum iteration count of 100,000 steps, uses RMSProp, and has a learning rate of 0.01 that decays exponentially.

Images