A method for detecting equipment instrument parameters based on deep learning and watershed segmentation

Abstract

This invention discloses a method for detecting instrument parameters based on deep learning and watershed segmentation, comprising: acquiring an original image dataset containing an instrument panel; constructing a deep convolutional neural network model to output the complete region of the instrument panel and two semantic segmentation prediction masks of the central region in the image; preprocessing the complete region of the instrument panel, the two semantic segmentation prediction masks of the central region, and the original image dataset to output the original image of the defined region and a label image; using a watershed algorithm with the help of the label image to segment the original image of the defined region, thereby locating each instrument in the image; taking a screenshot of each instrument region in the image where the instrument is located, and using an optical character recognition method to obtain instrument parameter information from each screenshot.

Description

Technical Field

[0001] This invention relates to the field of equipment and instrument testing technology, and in particular to a method for testing equipment and instrument parameters based on deep learning and watershed segmentation. Background Technology

[0002] The development of IoT and edge computing technologies has placed higher demands on the real-time acquisition capabilities of production equipment parameters. However, some equipment, due to its age, has poorly designed communication functions or is no longer in service, resulting in the inability to provide timely feedback of real-time parameters. For such equipment, network cameras can be used to detect and identify the instrument data parameters on the equipment panel, collect the data, and upload it.

[0003] Traditional device panel instrument data parameter recognition requires specifying a fixed area on the device panel and then identifying the instrument data within that area. However, this necessitates initializing each device panel and specifying a fixed area. This makes the method lack adaptability and generalization performance; when the camera experiences unpredictable erroneous displacement, the panel data cannot be recognized or read. Furthermore, devices often contain numerous, multi-scale, and multi-category instruments, and traditional cameras cannot simultaneously recognize different instrument data parameters. Therefore, this method treats device instrument parameter detection as detecting instrument data at arbitrary locations in a natural scene. That is, device panel instrument data can appear in any shape at any location within the camera's field of view, and it can simultaneously recognize numerous, multi-scale, and multi-category instrument data in natural scenes. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention discloses a method for detecting equipment instrument parameters based on deep learning and watershed segmentation, specifically including the following technical solutions:

[0005] Obtain the raw image dataset with the dashboard device;

[0006] A convolutional neural network semantic segmentation model is constructed. The original image dataset is input into the deep convolutional neural network model to perform image feature recognition, thereby outputting the complete area of ​​the dashboard device in the image and two semantic segmentation prediction masks of the central area.

[0007] Preprocess the complete region and the central region of the dashboard device using two semantic segmentation prediction masks and the original image dataset, and output the original image of the defined region and the labeled image.

[0008] The watershed algorithm is used to segment the original image of a limited region with the help of the labeled image, thereby locating each instrument in the image;

[0009] Each instrument area in the image used to locate the instrument is captured as a screenshot, and the instrument parameter information is obtained from each screenshot using optical character recognition (OCR).

[0010] Furthermore, the deep convolutional neural network model includes a backbone network, a deep feature fusion module, an extended receptive field module, and an output prediction mask module;

[0011] The backbone network extracts and recognizes semantic features of the original image dataset to obtain feature maps.

[0012] The enhanced feature fusion module performs upsampling fusion and downsampling fusion on the feature map and outputs an enhanced fused feature map;

[0013] The extended receptive field module uses convolution kernels with various dilation rates to perform dilated convolution on the enhanced fusion feature map, and then performs convolution and pooling fusion operations to output the final feature map;

[0014] The output prediction mask module obtains the target region mask and the target center region mask from the final feature map through full convolution and upsampling operations.

[0015] Furthermore, the deep convolutional neural network model is trained, and its parameters are adjusted using a loss function and backpropagation algorithm to ensure that the model's detection performance meets the requirements. The loss function of the model is shown below:

[0016]

[0017] In the formula: and The prediction errors for the target region mask and the target center region mask are respectively. A dice loss is used to supervise the prediction results, where λ is a parameter used to balance... and Since the two masks are considered equally important, λ is set to 1.

[0018] Furthermore, the preprocessing operations include denoising the image and generating a priori images;

[0019] When performing image denoising: the original image dataset is denoised using median filtering, and the two masks output from the semantic segmentation stage are processed using morphological opening operations, and finally the denoised target region mask is output.

[0020] When generating the prior image: the denoised target region mask and the original image are used to perform a Hadamard product operation to generate the original image of the defined region; the target region mask and the target center region mask are subtracted; and each connected component in the generated mask is marked to obtain a marked image.

[0021] Furthermore, when using the watershed algorithm to segment the original image of the defined region with the help of the label image: the label image and the original image of the defined region are used as prior knowledge for the watershed algorithm. Several positions in the label image are set as the initial iteration region, and the pixel height is determined according to the brightness value of the original image of the defined region. During the iteration process, the pixels with lower heights are gradually filled from the initial iteration region. During the process of iterating and filling different regions, a region boundary line will be formed. This boundary line is the boundary of the different instrument regions output.

[0022] Furthermore, the pixels in the labeled image are divided into three categories: uncertain regions of the instrument parameter area, non-instrument parameter areas, and instrument parameter center areas. Uncertain regions are generated by subtracting the instrument data center area mask from the instrument parameter area mask. Non-instrument parameter areas include pixels with a pixel value of 0 in the instrument area mask. The instrument parameter center area is the area with a pixel value of 1 in the instrument parameter center area mask.

[0023] By employing the aforementioned technical solution, this invention provides a method for detecting equipment instrument parameters based on deep learning and watershed segmentation. This method utilizes a camera to capture images and video information of old equipment instruments whose data cannot be exported, and then detects and identifies the instrument parameters to acquire them. When the camera is deployed near the equipment instrument, the method processes the camera's output image and automatically detects and locates the instrument parameters. During the recognition process, this method does not require manual specification of the specific location of the instrument parameters in the image or the hyperparameters of the algorithm model. In production, equipment often experiences vibrations and slight displacements. This causes changes in the relative position of the equipment instruments in the image. This method can still automatically detect the new position of the instruments in the image without human intervention. After completing the detection and location of the equipment instruments, this method uses a lightweight text recognition algorithm to quickly and accurately identify and output the instrument numbers. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of the method of the present invention;

[0026] Figure 2 This is a schematic diagram illustrating the process of constructing a convolutional neural network semantic segmentation model for this invention.

[0027] Figure 3 This is a schematic diagram of the image preprocessing operation process of the present invention. Detailed Implementation

[0028] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

[0029] like Figure 1 The present invention discloses a method for detecting equipment instrument parameters based on deep learning and watershed segmentation. The specific steps of the method are as follows:

[0030] S1: Acquire raw images of the device with the dashboard: Place the camera in front of the device from which you plan to collect instrument data, and adjust the focal length and image contrast appropriately. To eliminate flicker in the digital display, adjust the frame rate and shutter speed accordingly.

[0031] S2: Construct a convolutional neural network semantic segmentation model, the process is as follows: Figure 2 As shown:

[0032] S21: Constructing the backbone network. In this method, ResNet 18 is used as the backbone network for semantic feature extraction of images. ResNet 18 consists of 5 convolutional stages. This method uses the 4 different convolutional layers generated by the 2nd to 5th convolutional stages of ResNet 18 as the output feature maps of the backbone network.

[0033] S22: Construct a deepened feature fusion module, which is an improved pyramid feature network. The deepened feature fusion module consists of an upsampling fusion stage and a downsampling fusion stage. First, to match the input size of the module, the four sets of feature maps output from the backbone network are converted to 128 channels for each feature map using 1×1 convolutions. Second, these four sets of feature maps are input to the upsampling fusion stage, where high-level convolutional feature maps are upsampled level by level, and then the feature maps are fused level by level. Next, the merged feature maps are input to the downsampling fusion stage, where low-scale feature maps are downsampled level by level, and then the feature maps are fused sequentially. Finally, four sets of deepened fusion feature maps are output.

[0034] S23: Construct an extended receptive field module with four independent parallel branches, each with the same structure. The inputs of the four branches process four feature maps from the in-depth feature fusion module. Each branch performs three dilated convolutions on the input feature map, with dilation rates of 6%, 12%, and 18%, respectively. Simultaneously, the input feature map after average pooling is convolved with a 1x1 kernel. At this point, each branch outputs a total of five feature maps: three after dilated convolutions, one after average pooling, and one after a 1x1 convolution. Finally, the size of the five feature maps output by each branch is adjusted to match the size of the input feature maps by changing the kernel size and the 1x1 convolution.

[0035] S24: Constructing the output prediction mask module: Through two layers of fully convolutional networks and upsampling, the output is a final feature map with the same size as the original image and two channels. The output of the semantic segmentation network is two binary masks. One is the instrument region mask, where pixels within the instrument region have a value of 1, and pixels outside the instrument region have a value of 0. The other is the instrument center region mask, where pixels within the instrument center region have a value of 1, and pixels outside the instrument center region have a value of 0.

[0036] S25: Calculate the prediction error. In this method, after predicting the instrument area mask and the instrument center area mask, it is necessary to compare the true labels to update the neural network parameters in steps S21, S22, and S23. Therefore, the loss function of the model in this paper is shown in formula (1):

[0037]

[0038] In the formula: and These represent the prediction errors of the instrument area mask and the instrument center area mask, respectively. λ is a parameter used to balance... and Since the two masks are considered equally important, λ is set to 1 in all experiments.

[0039] Because the number of positive and negative sample pixels is extremely unbalanced in this task, dice loss is used to supervise the prediction results. and The specific formula is as follows:

[0040]

[0041]

[0042] Where P text (i) and P kernel (i) represents the i-th pixel in the prediction mask for the instrument region and the instrument core region, respectively. G text (i) and Gkernel (i) represents the i-th pixel in the real mask of the instrument region and the instrument core region, respectively. These four masks are binary images of the same size as the input image, where P... text and G text The meter pixel or P kernel and G kernel The kernel pixel in the instrument is 1; P text and G text Non-instrument pixels or P in kernel and G kernel The non-instrument kernel pixels are 0.

[0043] The true labels of the instrument region mask can generally be provided by the dataset, while the true labels of the instrument center region mask can be calculated from the true labels of the instrument region masks. This method uses the Vatti clipping algorithm to modify the original polygon P. n Perform d i Shrinking by one pixel yields the shrunken polygon P. i Subsequently, each shrunken polygon P i These are converted into binary masks, serving as the true labels for the central area masks of the instrumentation equipment. These true labels for the central area masks of the instrumentation equipment can be represented as G1, G2, ..., G... n We determine the appropriate shrinkage ratio as r. i ,P n and P i The difference d between i It can be calculated as:

[0044]

[0045] In the formula: Area(g) is the function for calculating the area of ​​the polygon, and Perimeter(g) is the function for calculating the perimeter of the polygon.

[0046] S3: Preprocess the complete region of the dashboard device and the two semantic segmentation prediction masks for the central region, as well as the original image dataset. The process is as follows: Figure 3 As shown:

[0047] S31: Image noise reduction. Before reconstructing the boundary of the instrument instance, the original input image and the two masks generated by the semantic segmentation network need to be preprocessed to eliminate noise. Median filtering is performed on the original image to remove noise. The calculation method of median filtering for two-dimensional images is shown in formula (3):

[0048] g(x,y)=med{f(xk,yl),(k,l∈W)} (5)

[0049] Where f(x,y) and g(x,y) are the original image and the processed image, respectively, med{·} represents the median operation, and W is the pixel range for median calculation, which is set to 21 in the method. To avoid incorrect segmentation at coarse contours by the watershed algorithm, morphological smoothing is performed on the two masks output from the semantic segmentation stage. The structuring element used in the morphological opening operation is a square matrix of length 3 with all elements being 1.

[0050] S32: Generate a priori image. To avoid the influence of non-instrument parameter regions on subsequent processing, a Hadamard product is performed between the instrument parameter region mask and the original image to generate the original image of the defined region. The labeled image provides initial values ​​and iteration regions for subsequent segmentation algorithms. The labeled image is generated by subtracting the target center region mask from the instrument data region mask and then labeling each connected component in the generated mask with a number. Pixels in the labeled image are divided into three categories: uncertain regions of the instrument parameter region, non-instrument parameter regions, and instrument parameter center regions. Uncertain segmented regions are generated by subtracting the instrument data center region mask from the instrument parameter region mask. Non-instrument parameter regions include pixels with a value of 0 in the instrument region mask. The instrument parameter center region is the region with a pixel value of 1 in the instrument parameter center region mask. In this paper, uncertain segmented regions are labeled as 0, non-instrument parameter regions are labeled as 1, and connected components in the instrument parameter center region are labeled with different numbers starting from 2.

[0051] S4: The watershed algorithm is used to segment the original image of the defined region using the label image. The label image and the original image of the defined region are used as prior knowledge for the watershed algorithm. Several positions in the label image are set as the initial iteration region, and the pixel height is determined according to the brightness value of the original image of the defined region. During the iteration process, pixels with lower heights are gradually filled from the initial iteration region. During the process of iterating and filling different regions, a boundary line will be formed, which is the boundary of the different instrument regions output. In order to obtain the edge information of the image, the gradient image in formula (6) is usually taken as the input image:

[0052]

[0053] g x =f(x,y)-f(x-1,y) (7)

[0054] g y =f(x,y)-f(x,y-1) (8)

[0055] Here, f(x,y) represents the original image, and grad[g] represents the gradient calculation. However, noise or small brightness variations in the image can easily lead to oversegmentation. Using prior knowledge to constrain the segmentation region can improve performance. Therefore, this method employs a watershed algorithm based on labeled images, using the labeled image and the foreground portion of the input image as prior knowledge for the watershed algorithm.

[0056] S5: Take a screenshot of each instrument area in the image where the instrument is located, and use optical character recognition (OCR) to obtain instrument parameter information from each screenshot:

[0057] S51: First, the maximum bounding rectangle is extracted sequentially from several non-connected Regions of Interest (ROIs) generated by the boundary reconstruction module.

[0058] S52: Set the pixels of the background portion of the outer rectangle to 0 to prevent the pixels in the background from affecting the recognition of instrument parameters.

[0059] S53: Input the processed screenshot into the Tesseract-ocr recognition module. Since the instrument output only involves numbers, the character candidate set should be set to only include numbers.

[0060] S54: Output the recognition results of each screenshot in order from top to bottom and from left to right according to their positions on the image.

[0061] Experimental verification shows that the present invention can effectively detect equipment and instrument parameters from natural scenes. It is a high-precision natural scene text detection model, and can overcome the problems of unstable detection ability and low robustness caused by arbitrary text arrangement and multi-scale text.

[0062] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A device instrument parameter detection method based on deep learning and watershed segmentation, characterized in that, The application relates to a method for instrument panel equipment parameter information acquisition. The method comprises the following steps: An original image dataset with instrument panel equipment is acquired; A convolutional neural network semantic segmentation model is constructed, original image dataset is input into a deep convolutional neural network model for image feature recognition, and two semantic segmentation prediction masks of complete regions and central regions of instrument panel equipment in pictures are output; Pretreatment operation is performed on the two semantic segmentation prediction masks of complete regions and central regions of instrument panel equipment and the original image dataset, and a limited region original image and a label image are output; A watershed algorithm is adopted to segment the limited region original image with the aid of the label image, so that each instrument in the image is positioned; Each instrument region in the image in which the instrument is positioned is captured, and an optical character recognition method is adopted to acquire instrument parameter information of each captured image; The deep convolutional neural network model comprises a backbone network, a deep feature fusion module, an extended receptive field module and an output prediction mask module; The backbone network extracts and recognizes picture semantic features of the original image dataset to obtain a feature map; The deep feature fusion module performs up-sampling fusion and down-sampling fusion on the feature map and outputs an enhanced fusion feature map; The extended receptive field module adopts convolution kernels with multiple hole rates to perform hole convolution on the enhanced fusion feature map, and then performs convolution and pooling fusion operation to output a final feature map; 2. The method of claim 1, wherein: The output prediction mask module acquires a target region mask and a target central region mask through full convolution and up-sampling operation on the final feature map. （1） In the formula: and are the prediction errors of the target region mask and the target center region mask, respectively, supervised by the dice loss to supervise the prediction results, for balancing and , is set to 1.

3. The method of claim 1, wherein: The deep convolutional neural network model is trained, the parameters of the model are adjusted by using a loss function and a back propagation algorithm, the detection performance of the model is controlled to meet the requirements, and the loss function of the model is shown as follows: The pretreatment operation comprises picture denoising and prior picture generation; During the picture denoising, the original image dataset is subjected to denoising treatment by adopting a median filter mode, the two masks output in the semantic segmentation stage are subjected to morphological opening operation treatment, and finally a denoised target region mask is output; 4. The method of claim 1, wherein: During the prior picture generation, the denoised target region mask and the original image are subjected to Hadamard product operation to generate a limited region original image, the target region mask and the target central region mask are subjected to subtraction operation, and each connected domain in the generated mask is marked to obtain a label image. During the segmentation of the limited region original image by adopting the watershed algorithm with the aid of the label image, the label image and the limited region original image are taken as prior knowledge of the watershed algorithm, a plurality of positions in the label image are set as initial iteration regions, the height of a pixel point is determined according to the brightness value of the limited region original image, in the iteration process, the initial iteration regions are gradually filled with pixel points with lower height, and in the process of iteration and filling of different regions, a region boundary line is formed, and the region boundary line is the output different instrument region boundary.

5. The method of claim 3, wherein: The pixels in the marked image are divided into three categories: an uncertain region of the instrument parameter region, a non-instrument parameter region and an instrument parameter central region, the uncertain region is generated by subtracting the instrument data central region mask from the instrument parameter region mask, the non-instrument parameter region includes the pixels with a pixel value of 0 in the instrument region mask, and the instrument parameter central region is a region with a pixel value of 1 in the instrument parameter central region mask.

Images