Industrial pointer instrument detection reading recognition method, device, equipment and medium

Abstract

The present invention provides an industrial pointer instrument detection reading recognition method, device, equipment and medium, including: obtaining an image to be detected; extracting a multi-scale first feature map from the image to be detected through a target detection model, constructing multi-scale high-level semantic feature information based on the multi-scale first feature map, and performing layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract the instrument image from the image to be detected; extracting a multi-scale second feature map from the instrument image through a semantic segmentation model, performing local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image; based on the pointer and scale mask image, determining the instrument reading result corresponding to the image to be detected. The present invention can significantly improve the accuracy of industrial pointer instrument detection reading recognition, thereby improving the safe use of different instruments in industrial scenes and ensuring that industrial production can be carried out stably.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial inspection, and in particular to an industrial pointer instrument detection reading recognition method, device, equipment and medium. Background Art

[0002] With the development of the times, various industries have continuously increased their requirements for the level of automation and the degree of machine intelligence. Under such requirements of the times, digital image processing technology has gradually been introduced into industrial automation and has become an increasingly important component.

[0003] In industrial production, machine learning is increasingly being used in instrument detection and identification methods, making instrument detection and identification methods advance rapidly in the direction of intelligence. Since the display of pointer instruments is different from digital signals and cannot be read by computers, data recording and pointer calibration are all done manually. Due to unavoidable external factors such as human factors, the reading accuracy is not high and the volatility is large, which greatly reduces the accuracy of data reading; on the other hand, the environment in which some instruments are installed is relatively harsh and not suitable for manual reading. Instrument detection and reading recognition is based on computer vision analysis technology, which can realize rapid detection and reading recognition of industrial instruments. The detection and reading recognition of intelligent pointer instruments can reduce the errors and omissions caused by manual readings while reducing the labor costs of enterprises. In addition, compared with manual inspections, the detection and reading recognition of intelligent pointer instruments can display and report the detection and recognition results more timely.

[0004] Therefore, compared with manual interpretation, machine vision equipment can run stably for a long time and has strong environmental adaptability, and can read the readings of pointer instruments in different environments. Pointer instrument detection and automatic reading solutions based on machine vision, including instrument target detection, tilt correction, instrument dial scale segmentation, reading display and other processes, can help realize unmanned intelligent inspection technology in industrial scenarios. However, the accuracy of pointer instrument detection and automatic reading solutions based on machine vision provided by related technologies still has room for improvement. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an industrial pointer instrument detection reading recognition method, device, equipment and medium, which can significantly improve the accuracy of industrial pointer instrument detection reading recognition, thereby improving the safe use of different instruments in industrial scenarios and ensuring that industrial production can proceed stably.

[0006] In a first aspect, an embodiment of the present invention provides a method for identifying a reading of an industrial pointer instrument, comprising:

[0007] Acquire the image to be detected;

[0008] Extracting a multi-scale first feature map from the image to be detected by using a pre-trained target detection model, constructing multi-scale high-level semantic feature information based on the multi-scale first feature map, and performing layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract an instrument image from the image to be detected;

[0009] Extracting a multi-scale second feature map from the instrument image through a pre-trained semantic segmentation model, and performing local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image;

[0010] Based on the pointer and scale mask image, the instrument reading result corresponding to the image to be detected is determined.

[0011] In one embodiment, the target detection model includes a backbone network, the backbone network includes a Focus unit, a feature extraction unit and an SPP unit, and the feature extraction units each include an alternately connected CBL sub-network and a large selection kernel sub-network;

[0012] The step of extracting a multi-scale first feature map from the image to be detected includes:

[0013] By means of the Focus unit, pixel points are periodically extracted from the high-resolution image to be detected and reconstructed into a low-resolution image;

[0014] In the feature extraction unit, after the CBL sub-network processes the low-resolution image, the input feature is extracted through the large selection kernel sub-network, and the feature output by the large selection kernel sub-network is processed through the next CBL sub-network, and this process is repeated until the last CBL sub-network outputs a feature;

[0015] Through the SPP unit, the features output by the last CBL sub-network are pooled and fused;

[0016] The features output by the large selection kernel network specified in the feature extraction unit and the output features of the SPP unit are used as a multi-scale first feature map.

[0017] In one embodiment, the target detection model further includes a feature pyramid network, which includes a plurality of large selection kernel networks and a plurality of first feature processing networks from bottom to top;

[0018] The step of constructing multi-scale high-level semantic feature information based on the multi-scale first feature map includes:

[0019] Inputting the first feature map into the large selection kernel network in the feature pyramid network, and performing feature extraction on the input features through the large selection kernel network;

[0020] In the first feature processing network, after the features output by the large selection kernel subnetwork are processed by the CBL subnetwork in the first feature processing network, the features output by the CBL subnetwork are upsampled, and the upsampled features are fused with the first feature map having the same size as the features;

[0021] Inputting the fused features into the next large selection kernel network in the feature pyramid network, and extracting the input features through the large selection kernel network;

[0022] In the next first feature processing network, the features output by the large selection kernel network are upsampled, and the upsampled features are fused with the first feature map of the same size to obtain multi-scale high-level semantic feature information.

[0023] In one embodiment, the target detection model further includes a channel aggregation network, which includes a plurality of top-down large selection kernel networks and a plurality of second feature processing networks;

[0024] The step of performing layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract the instrument image from the image to be detected includes:

[0025] Inputting the high-level semantic feature information into the large selection kernel network in the channel aggregation network, and extracting the input features through the large selection kernel network;

[0026] In the second feature processing network, after the features output by the large selection kernel sub-network are processed by the CBL sub-network in the first feature processing network, the features output by the CBL sub-network are fused with the high-level semantic feature information having the same size as the features output by the CBL sub-network;

[0027] Continue to input the fused features into the next large selection nuclear sub-network in the channel aggregation network, and repeat this process until the last large selection nuclear sub-network in the channel aggregation network outputs features;

[0028] An instrument image is generated based on features output by each of the large selection kernel sub-networks in the channel aggregation network.

[0029] In one embodiment, the step of extracting features from input features by using the large selection kernel network includes:

[0030] Use the first convolution kernel to perform a convolution operation on the input features;

[0031] Using a second convolution kernel to perform a convolution operation on the convolution operation result of the first convolution kernel;

[0032] Concatenate the convolution operation result of the first convolution kernel and the convolution operation result of the second convolution kernel;

[0033] After Avgpool, convolution, and sigmoid operations, the concat result is subjected to dot multiplication operations with the convolution operation results of the first convolution kernel and the convolution operation results of the second convolution kernel respectively;

[0034] The dot product results are concat-concatenated, and the concat result is dot-multiplied with the input features before output.

[0035] In one embodiment, the semantic segmentation model includes a hierarchical Transformer encoder and a lightweight All-MLP decoder, the hierarchical Transformer encoder includes a plurality of Transform units, each of the Transform units includes a plurality of feature extraction subunits and merging subunits, each of the feature extraction subunits includes an attention layer and a PAN-FFN network;

[0036] The step of extracting a multi-scale second feature map from the instrument image and performing local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image includes:

[0037] For each Transform unit, the input instrument image or the features output by the previous Transform unit are processed by the attention layer in the Transform unit, and the features output by the attention layer are extracted by the PAN-FFN network in the Transform unit to obtain a second feature map output by the Transform unit;

[0038] The lightweight All-MLP decoder performs local and global attention processing on the second feature map output by each Transform unit to generate a pointer and scale mask image corresponding to the instrument image.

[0039] In one embodiment, the step of extracting features of the output of the attention layer through the PAN-FFN network in the Transform unit includes:

[0040] Using a third convolution kernel to perform a convolution operation on the features output by the attention layer, and using a Dw-Conv layer to perform a convolution operation on the features output by the attention layer; wherein the step size of the third convolution kernel is consistent with the step size of the Dw-Conv layer;

[0041] Fusion the convolution result of the third convolution kernel with the convolution result of the Dw-Con;

[0042] Using a fourth convolution kernel to perform a convolution operation on the fused features, and using a Dw-D-Conv layer to perform a convolution operation on the convolution result of the Dw-Conv layer; wherein the step size of the fourth convolution kernel is consistent with the step size of the Dw-D-Conv layer;

[0043] Fusion the convolution result of the fourth convolution kernel and the convolution result of the Dw-D-Conv;

[0044] Using the fifth convolution kernel to perform a convolution operation on the fused features, and using the sixth convolution kernel to perform a convolution operation on the convolution result of the Dw-D-Conv;

[0045] The convolution result of the fifth convolution kernel is concatenated with the convolution result of the sixth convolution kernel and then output.

[0046] In one embodiment, the step of determining the instrument reading result corresponding to the image to be detected based on the pointer and scale mask image includes:

[0047] Performing image corrosion processing on the pointer and scale mask images;

[0048] Expanding the pointer and scale mask image after the image corrosion process into a rectangular image;

[0049] Based on the position of the positioning pointer relative to the scale in the rectangular image, and based on the rectangular image to determine the dial range, the instrument reading result corresponding to the image to be detected is determined according to the position and the dial range.

[0050] In a second aspect, an embodiment of the present invention further provides an industrial pointer instrument detection reading recognition device, comprising:

[0051] An image acquisition module, used for acquiring an image to be detected;

[0052] The target detection module is used to extract a multi-scale first feature map from the image to be detected by using a pre-trained target detection model, construct multi-scale high-level semantic feature information based on the multi-scale first feature map, and perform layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract an instrument image from the image to be detected;

[0053] A semantic segmentation module, configured to extract a multi-scale second feature map from the instrument image by using a pre-trained semantic segmentation model, and perform local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image;

[0054] The instrument reading module is used to determine the instrument reading result corresponding to the image to be detected based on the pointer and scale mask image.

[0055] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0056] An industrial pointer instrument detection reading recognition method, device, equipment and medium provided by an embodiment of the present invention first obtains an image to be detected; then, a multi-scale first feature map is extracted from the image to be detected through a pre-trained target detection model, multi-scale high-level semantic feature information is constructed based on the multi-scale first feature map, and a layer-by-layer feature fusion process is performed on the multi-scale high-level semantic feature information to extract the instrument image from the image to be detected; then, a multi-scale second feature map is extracted from the instrument image through a pre-trained semantic segmentation model, and local and global attention processing is performed on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image; finally, based on the pointer and scale mask image, the instrument reading result corresponding to the image to be detected is determined. The above method extracts the multi-scale first feature of the image to be detected through the target detection model, and further constructs multi-scale high-level semantic feature information, and then performs layer-by-layer feature fusion to extract the instrument image from the image to be detected. On this basis, the semantic segmentation model is further used to extract the multi-scale second feature map from the instrument image, and the corresponding pointer and scale mask images are generated through local and global attention processing. The mask image is post-processed to obtain the final instrument reading result. The embodiment of the present invention uses the above-mentioned target detection model and semantic segmentation model to significantly improve the accuracy of industrial pointer instrument detection reading recognition, thereby improving the safe use of different instruments in industrial scenes and ensuring that industrial production can proceed stably.

[0057] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of a flow chart of a method for detecting and identifying readings of an industrial pointer instrument provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of the structure of a target detection algorithm YOLOv5s provided in an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of the structure of a large selection kernel network provided by an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of the principle of a Shape-IoU loss function provided by an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of the structure of a semantic segmentation network structure SegFormer provided in an embodiment of the present invention;

[0065] Figure 6 A schematic diagram of the structure of a PAN-FFN network provided in an embodiment of the present invention;

[0066] Figure 7 A schematic diagram of an instrument reading result provided by an embodiment of the present invention;

[0067] Figure 8 A schematic diagram of another instrument reading result provided by an embodiment of the present invention;

[0068] Fig. 9 An overall flow chart of an industrial pointer instrument detection reading recognition method provided by an embodiment of the present invention;

[0069] Fig.10 A schematic diagram of the structure of an industrial pointer instrument detection and reading recognition device provided by an embodiment of the present invention;

[0070] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] At present, there is still room for improvement in the accuracy of the machine vision-based pointer instrument detection and automatic reading solutions provided by related technologies. Based on this, the present invention implements a method, device, equipment and medium for industrial pointer instrument detection and reading recognition, which can significantly improve the accuracy of industrial pointer instrument detection and reading recognition, thereby improving the safe use of different instruments in industrial scenarios and ensuring that industrial production can proceed stably.

[0073] To facilitate understanding of this embodiment, firstly, a method for detecting and identifying readings of an industrial pointer instrument disclosed in an embodiment of the present invention is described in detail. Figure 1 The flowchart of a method for detecting and identifying readings of an industrial pointer instrument is shown, and the method mainly includes the following steps S102 to S108:

[0074] Step S102, obtaining an image to be detected. The content displayed in the image to be detected includes at least instruments. In one example, an inspection device (such as a robot or a drone, etc.) can be used to collect images of instruments in a factory to obtain the image to be detected.

[0075] Step S104, extracting a multi-scale first feature map from the image to be detected through the pre-trained target detection model, constructing multi-scale high-level semantic feature information based on the multi-scale first feature map, and performing layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract the instrument image from the image to be detected.

[0076] In one example, the target detection model can adopt an improved YOLOv5s network, and the target detection model includes a backbone network, a feature pyramid network (FPN) and a channel aggregation network (PAN). The backbone network is used to extract a multi-scale first feature map from the image to be detected, the feature pyramid network (FPN) is used to construct multi-scale high-level semantic feature information based on the multi-scale first feature map, and the channel aggregation network (PAN) performs layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract the instrument image from the image to be detected.

[0077] Step S106, extracting a multi-scale second feature map from the instrument image through the pre-trained semantic segmentation model, performing local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image.

[0078] In one example, the semantic segmentation model can adopt an improved Segformer network, which includes a hierarchical Transformer encoder and a lightweight All-MLP decoder. The hierarchical Transformer encoder is used to extract multi-scale second feature maps from the instrument image, and the lightweight All-MLP decoder is used to perform local and global attention processing on the multi-scale second feature maps to generate pointer and scale mask images corresponding to the instrument image.

[0079] Step S108, based on the pointer and scale mask image, determine the instrument reading result corresponding to the image to be detected.

[0080] In one example, the image erosion technology is used to reduce noise interference in the pointer and scale mask images segmented by the semantic segmentation model; then the circular pointer segmentation outline is expanded into a rectangular image through coordinate transformation; finally, the relative scale position of the pointer is located according to the threshold in the rectangular image, and the dial type is determined according to the number of dial scale lines to obtain the dial range, thereby calculating the instrument reading result.

[0081] The above method provided by the embodiment of the present invention extracts the multi-scale first feature of the image to be detected through the target detection model, and further constructs multi-scale high-level semantic feature information, and then performs layer-by-layer feature fusion to extract the instrument image from the image to be detected, and on this basis continues to use the semantic segmentation model to extract the multi-scale second feature map from the instrument image, and generates the corresponding pointer and scale mask image through local and global attention processing, and post-processes the mask image to obtain the final instrument reading result. The embodiment of the present invention uses the above-mentioned target detection model and semantic segmentation model to significantly improve the accuracy of industrial pointer instrument detection reading recognition, thereby improving the safe use of different instruments in industrial scenes and ensuring that industrial production can proceed stably.

[0082] For ease of understanding, the present invention provides a specific implementation of an industrial pointer instrument detection reading recognition method, including:

[0083] (1) Sample selection: Since instrument detection reading recognition is used in different scenes in the factory, it is necessary to select images with different brightness as the training set of the instrument detection model. At the same time, it is necessary to perform semantic segmentation on the instrument dial and pointer, so the minimum semantic segmentation pixel is not less than 5×5 pixels;

[0084] (2) Dataset production: After selecting samples, they are manually labeled. In order to make the original data more diverse, larger images are randomly cropped, and the cropping range is 1 / 8 to 1 / 4 of the original image. The randomly cropped images are spliced. First, the size can be adapted to the input size of the deep learning network as much as possible, and the diversity of image features learned by the algorithm. Second, random splicing can increase the number of image samples, which can achieve the expected results, reduce training time to a certain extent, and avoid model overfitting.

[0085] (3) Data preprocessing: Data preprocessing is a key factor in solving practical problems and improving algorithm accuracy in deep learning algorithms. In industrial production environments, due to factors such as light and the environment, there will inevitably be interference such as reflections and dust on the instrument dials, which may lead to missed detections during the detection of the instrument's large scale lines, thereby affecting the accuracy of subsequent readings of pointer-type instrument readings.

[0086] Based on this, the embodiment of the present invention first uses instruments with clearer scales on the dashboard as annotation data for semantic segmentation, and then performs basic data enhancement operations such as rotation, flipping, saturation, brightness, translation, motion blur and Gaussian blur on the original data to solve the problem of diversity in factory environments. The positive sample filling method is then used to randomly add instrument images to the negative sample images to increase the number of positive samples, so that the positive and negative samples of the data remain relatively balanced.

[0087] (4) Selection of benchmark network model: Currently, there are three main algorithms for target semantic segmentation: traditional algorithms based on machine vision, convolutional neural network algorithms based on deep learning (CNN), and self-attention networks of the Transformer series.

[0088] (a) Traditional algorithms based on machine vision: They mainly use template matching, feature matching, Hough transform, and region growing techniques to locate targets. They have relatively high accuracy, but are difficult to solve complex problems.

[0089] (b) Based on deep learning CNN convolutional neural network algorithm: The convolutional neural network algorithm shares convolution kernels, can process high-dimensional data, and can automatically extract features. However, when the number of network layers is too deep, the number of parameters is large, so the CNN convolutional neural network detection network needs to be optimized.

[0090] (c) The Transformer series of network algorithms introduces the self-attention mechanism in natural language processing into computer vision.

[0091] First, the global characteristics: The CNN algorithm has the problem of limited local receptive field. In order to obtain global information, multiple layers need to be stacked, but as the number of layers increases, the amount of information will be exhausted, so the extracted features are focused on certain areas. Transformer has a self-attention mechanism that can effectively obtain global information, and multiple heads can map it to multiple spaces, making the model more expressive.

[0092] Secondly, Transformer has a good modal fusion capability: for images, the initial embeddings obtained by convolution or direct pixel operation can be fed into Transformer without always maintaining the feature map structure of the channel. For example, position embeddings can be easily used as long as the information can be encoded.

[0093] Finally, Multiple Tasks capability: A Transformer can perform many tasks because its Attention mechanism allows the network to learn different tasks differently. A simple usage is to add an Embedding of a Task ID.

[0094] However, the Transformer series network algorithm has low computational efficiency. Since the mechanism is a set of Transformer structures in NLP, model training requires a large number of samples and occupies a large amount of training resources. In addition, the Transformer series network algorithm ignores the two-dimensional structural information of the image itself, only considers spatial adaptability, and ignores adaptability in channel dimension.

[0095] Instrument detection and segmentation belong to two-dimensional information. The two-dimensional structural information of the image itself is very critical, and the spatial adaptability and channel adaptability determine the effect of instrument detection and segmentation. Instrument detection and reading recognition realize the detection and reading recognition of pointer instruments through instrument detection, semantic segmentation, image tilt correction, image corrosion, image binarization, etc. In the detection and recognition process, it is necessary not only to reduce the limitation of data volume, the complexity of the model, and the depth of the number of network layers, but also to improve the operation efficiency and accuracy of the algorithm. Therefore, the embodiment of the present invention selects an algorithm mechanism that combines the target detection model of the CNN network algorithm with the semantic segmentation network algorithm of the Transformer series, which not only meets the high efficiency of the algorithm, but also meets the accurate detection and reading recognition of pointer instruments.

[0096] (5) Explain the target detection algorithm YOLOv5s: Build the target detection network structure YOLOV5, see Figure 2The structure diagram of a target detection algorithm YOLOv5s is shown in the figure. The algorithm proposes a top-down pathway and a bottom-up pathway in the Neck part, which is a combination of FPN+PAN. The Feature Pyramid Network (FPN) is used to process the outputs from multiple scales in the Backbone, and then construct high-level semantic feature information. Specifically, the Feature Pyramid Network (FPN) is usually composed of multiple convolutional layers and pooling layers of different sizes, which are used to generate feature maps of different scales. Feature maps of different scales can provide target information of different granularities. The Path Aggregation Network (PAN) compensates and fuses strong positioning information through a bottom-up structure, and finally uses the output for Head detection. Specifically, the feature fusion module in the PAN uses a top-down feature aggregation method to fuse features layer by layer, thereby forming a more powerful feature representation. It can enhance the extraction of feature information in the target detection process, thereby effectively improving the performance of target detection.

[0097] for Figure 2 The target detection algorithm YOLOv5s provided in the embodiment of the present invention is improved. The traditional YOLOV5 adopts the CSPDarknet53 backbone network for feature extraction. CSPDarknet53 includes the C3 module. The BottleNeck module in the C3 module is obtained by stacking residual modules. In the traditional convolutional neural network, the size of the convolution kernel is fixed, and the global information of the image cannot be fully obtained. Therefore, it is proposed to introduce the C3_LSKNet module to replace the BottleNeck module in the C3 module with a large selection kernel network (LSKNet). Specifically, the target detection model includes a backbone network, a feature pyramid network (FPN), and a channel aggregation network (PAN). The backbone network includes a Focus unit, a feature extraction unit and an SPP unit. The feature extraction units include alternately connected CBL sub-networks and large selection kernel networks (LSKNet); the target detection model also includes a feature pyramid network, which includes multiple large selection kernel networks (LSKNet) from bottom to top and multiple first feature processing networks; the target detection model also includes a channel aggregation network, which includes multiple large selection kernel networks (LSKNet) from top to bottom and multiple second feature processing networks.

[0098] In combination with the aforementioned network structure, when performing the step of extracting a multi-scale first feature map from the image to be detected, the following steps 1A to 1D may be referred to:

[0099] In step 1A, the Focus unit is used to periodically extract pixels from the high-resolution image to be detected and reconstruct them into a low-resolution image. That is, the four adjacent positions of the image are stacked, and the wh width and height dimensional information is focused on the c channel space. This can improve the receptive field of each point, reduce the loss of original information, reduce the amount of calculation, and speed up the operation.

[0100] Step 1B, in the feature extraction unit, after the low-resolution image is processed by the CBL sub-network, the input features are extracted by the large selection kernel network (LSKNet), and the features output by the large selection kernel network (LSKNet) are processed by the next CBL sub-network, and this process is repeated until the last CBL sub-network outputs features;

[0101] Step 1C: Through the SPP unit, the features output by the last CBL sub-network are pooled and fused. Specifically, the input feature maps are pooled with different sizes, and then the pooling results of different scales are fused through convolution operations. The feature information of different scales can be adaptively fused, thereby enhancing the feature expression and perception capabilities of the model.

[0102] Step 1D: The features output by the large selection kernel network (LSKNet) specified in the feature extraction unit and the features output by the SPP unit are used as the first multi-scale feature map. Figure 2 It can be seen that the features output by the second large selection kernel network (LSKNet), the features output by the third large selection kernel network (LSKNet), and the features output by the SPP unit can be used as the first multi-scale feature map.

[0103] In combination with the aforementioned network structure, when executing the step of constructing multi-scale high-level semantic feature information based on the multi-scale first feature map, the following steps 2A to 2D may be referred to:

[0104] Step 2A, input the first feature map into the large selection kernel network in the feature pyramid network (FPN), and extract the input features through the large selection kernel network (LSKNet). Specifically, the features output by the SPP unit are input into the large selection kernel network (LSKNet) in the feature pyramid network (FPN), and the input features are extracted through the large selection kernel network (LSKNet).

[0105] Step 2B, in the first feature processing network, after the features output by the large selection kernel network (LSKNet) are processed by the CBL sub-network in the first feature processing network, the features output by the CBL sub-network are upsampled through the upsampling layer (upsample), and the upsampled features are fused with the first feature map of the same size through the connection layer (concat), specifically, the upsampled features are fused with the features output by the third large selection kernel network (LSKNet).

[0106] In step 2C, the fused features are input into the next large selection kernel network (LSKNet) in the feature pyramid network, and the input features are extracted through the large selection kernel network (LSKNet).

[0107] Step 2D, in the next first feature processing network, the features output by the large selection kernel network (LSKNet) are upsampled through the upsampling layer (upsample), and the upsampled features are fused with the first feature map of the same size through the connection layer (concat) to obtain multi-scale high-level semantic feature information, specifically, the upsampled features are fused with the features output by the second large selection kernel network.

[0108] In combination with the aforementioned network structure, when performing a layer-by-layer feature fusion process on multi-scale high-level semantic feature information to extract the instrument image from the image to be detected, the following steps 3A to 3D may be referred to:

[0109] Step 3A, input the high-level semantic feature information into the large selection kernel network (LSKNet) in the channel aggregation network (PAN), and extract the input features through the large selection kernel network (LSKNet). Specifically, the features output by the connection layer (concat) in the feature pyramid network (FPN) are input into the large selection kernel network (LSKNet) in the channel aggregation network (PAN), and the input features are extracted through the large selection kernel network (LSKNet).

[0110] Step 3B, in the second feature processing network, after the features output by the large selection kernel network (LSKNet) are processed by the CBL sub-network in the first feature processing network, the features output by the CBL sub-network are fused with the high-level semantic feature information of the same size through the connection layer (concat), specifically, the features output by the CBL sub-network are fused with the features output by the large selection kernel network (LSKNet) in the feature pyramid network (FPN).

[0111] Step 3C, continue to input the fused features into the next large selection kernel network (LSKNet) in the channel aggregation network (PAN), and repeat this process until the last large selection kernel network (LSKNet) in the channel aggregation network (PAN) outputs features. Figure 2 It can be seen that the fused features will be input into the next large selection kernel network (LSKNet), and the processed features will be processed by the CBL sub-network and fused with the features output by the CBL sub-network in the feature pyramid network (FPN). The fused features will be input into the last large selection kernel network (LSKNet), and the features will be output by the last large selection kernel network (LSKNet).

[0112] Step 3D, generates an instrument image based on the features output by each large selection kernel network in the channel aggregation network.

[0113] Further, the embodiment of the present invention explains the large selection kernel network (LSKNet) provided in the above embodiment. The large selection kernel network (LSKNet) can dynamically adjust the receptive field of the feature extraction backbone to more effectively handle the different wide backgrounds of the detected object. In the module, the size of the convolution kernel is dynamically determined according to the input, which means that the model can adjust the receptive field of each target as needed to better capture the characteristics of the target. Specifically, it includes: using the first convolution kernel to perform a convolution operation on the input feature; using the second convolution kernel to perform a convolution operation on the convolution operation result of the first convolution kernel; concatenating the convolution operation result of the first convolution kernel with the convolution operation result of the second convolution kernel; performing a dot multiplication operation on the concat splicing result after Avgpool, convolution, and sigmoid operations with the concat splicing result of the first convolution kernel and the convolution operation result of the second convolution kernel; concatenating the dot multiplication result, and outputting the concat splicing after performing a dot multiplication operation with the input feature.

[0114] In a specific implementation, the first convolution kernel may be a 5×5 convolution kernel, and the second convolution kernel may be a 7×7 convolution kernel. Figure 3 The schematic diagram of the structure of a large selection kernel network shown in the figure includes: a 5×5 convolution kernel (Conv), a 7×7 convolution kernel (Conv), a connection layer (concat), an average pooling layer (Avgpool), a convolution and activation function layer (Conv+sigmoid), and a connection layer (concat).

[0115] On this basis, for the input of each layer, first use a 5×5 convolution kernel to perform a convolution operation, then convolve the result of the 5×5 convolution kernel through a 7×7 convolution kernel, and then concatenate the result of the 5×5 convolution kernel with the result of the 7×7 convolution kernel. The concatenated result is subjected to Avgpool, convolution, and sigmoid operations, and the output result is respectively multiplied by the convolution operation result of the 5×5 convolution kernel and the convolution operation result of the 7×7 convolution kernel, and then the result is concat-joined. Finally, the concatenation result is multiplied by the input feature and output. The overall process is a complete C3_LSKNet structure, which realizes the overall process of the C3_LSKNet algorithm module.

[0116] Furthermore, when training the target detection algorithm YOLOv5s, the Shape-IoU loss function is used to replace the original CIoU:

[0117] Use the Shape-IoU loss function to replace the original CIoU. The original CIoU loss function does not consider the direction between the real box and the predicted box, resulting in a slow convergence speed. The original CIoU only considers the impact of the distance, shape, and angle between the GT box and the Anchor box on the bounding box regression, but ignores the impact of the shape and size of the bounding box itself on the bounding box regression. The original CIoU loss function has the following problems:

[0118] 1) Assuming that the GT box is not a square but has long and short sides, the differences in bounding box shape and size in the regression samples will lead to differences in their IoU values ​​when the bias and shape deviation are the same.

[0119] 2) For bounding box regression samples of the same scale, when the bias and shape bias are the same and not 0, the shape of the bounding box will affect the IoU value of the regression sample. The change in the bias of the bounding box in the short side direction and the corresponding change in the shape bias have a more significant impact on the IoU value.

[0120] 3) For regression samples with the same shape, when the bias and shape bias of the regression samples are the same and not 0, the IoU value of the smaller scale bounding box regression sample is more significantly affected by the GT box shape than the larger scale regression sample.

[0121] The existing Shape-IoU loss function can eliminate the influence of the shape and size of the bounding box regression sample itself on the bounding box regression, see Figure 4 The schematic diagram of the principle of a Shape-IoU loss function is shown in Figure 1. Figure 4 The two shaded areas in the figure are the predicted instrument detection box B and the actual instrument detection box B. gt, Shape-IoU loss function, specifically includes:

[0122]

[0123]

[0124] Among them, B is the predicted instrument detection box, B gt is the real instrument detection frame, w gt is the width of the actual instrument detection frame, h gt is the length of the real instrument detection box, scale is the scaling factor, which is related to the size of the target in the dataset, ww and hh represent the weight coefficients in the horizontal and vertical directions respectively, and their values ​​are related to the shape of the GT box, and distance shape为 Center point distance, x c ,y c is the center point coordinate of the predicted instrument detection frame, is the center point coordinate of the real instrument detection frame, c is the sum of the squares of the center point differences, Ω shape is the shape loss, and θ is the control of the attention paid to the shape loss. The corresponding bounding box regression loss L Shape-IoU as follows:

[0125] L Shape-IoU＝ 1-IoU+disatnce shape +0.5×Ω shape .

[0126] Preferably, before executing the aforementioned step S106, the instrument image output by the target detection model can be cropped, and after the cropped instrument image is tilt-corrected, the tilt-corrected instrument image is input into the semantic segmentation model. The position of the instrument target can be obtained through target detection, but since it is difficult to ensure that the camera lens plane and the instrument plane are parallel, the acquired instrument image usually has a certain tilt angle or a certain degree of deformation, which has a great impact on the subsequent readings. Therefore, it is necessary to correct the image to convert the irregular instrument into a regular instrument, and use the perspective transformation method to correct the instrument dial.

[0127] (6) Explanation of the semantic segmentation algorithm Segformer: Semantic segmentation is to classify different types of objects in an image. When the instrument panel area is cropped after target detection, the interference of the external environment is also removed, but the instrument panel still has the influence of characters, etc., and semantic segmentation technology can effectively remove the internal interference of the dial, so the SegFormer model is used to segment the instrument panel and pointer.

[0128] Build the semantic segmentation network structure SegFormer, see Figure 5The diagram shows the structure of a semantic segmentation network structure SegFormer. The algorithm proposes a simple, efficient, but powerful semantic segmentation framework that combines a hierarchical Transformer encoder and a lightweight All-MLP decoder. The hierarchical Transformer encoder is used to generate high-resolution coarse features and low-resolution fine features. The hierarchical Transformer encoder can output multi-scale features, but does not require position encoding, so there is no need to interpolate the position encoding; the lightweight All-MLP decoder fuses these multi-level features to generate the final semantic segmentation mask. The lightweight All-MLP decoder aggregates information from different layers, thereby combining local attention and global attention to present a powerful representation, which can express more comprehensive feature information.

[0129] The SegFormer model has the following advantages:

[0130] (I) Hierarchical feature representation. The goal of this module is to generate multi-level features similar to CNN given an input image. These features provide high-resolution coarse features and low-resolution fine features, which can usually improve the performance of semantic segmentation. Given an input image with a resolution of H×W×C, the Transformer Block module can generate Where i ranges from 1 to 4, C i+1 Than C i big;

[0131] (ii) Efficient self-attention. The main computational bottleneck of the encoder is the self-attention layer. In the original multi-head self-attention process, each head Q, K, V has the same dimension N×C, where N=H×W is the length of the sequence, and the calculation of self-attention is:

[0132]

[0133] The computational complexity of this process is O(N 2 ).

[0134]

[0135] Segformer uses the reduction ratio R to shorten the sequence length, making the new output dimension The complexity of self-attention changes from O(N 2 ) will be The running speed of the model is greatly improved, where K is the sequence to be reduced.

[0136] (iii) Mix-FFN. Mix-FFN mixes a 3×3 convolution and an MLP into each FFN, while using depthwise convolution to reduce the number of parameters and improve efficiency.

[0137] for Figure 5 The semantic segmentation network structure SegFormer is provided. The embodiment of the present invention improves the semantic segmentation network structure SegFormer. The Mix-FFN module feeds the feature maps obtained from different layers into a feedforward network and restores them to 1 / 4 of the original image by upsampling. The high-resolution coarse features and the low-resolution fine features do not interact with each other. Some detail information will be lost in the Mix-FFN process. Therefore, it is proposed to introduce a PAN-FFN network. That is, the hierarchical Transformer encoder includes multiple Transform units, each of which includes multiple feature extraction subunits and merging subunits, and each feature extraction subunit includes an attention layer and a PAN-FFN network.

[0138] Regarding the aforementioned step S106, the embodiment of the present invention first explains the semantic segmentation model.

[0139] In combination with the aforementioned network structure, when performing the steps of extracting a multi-scale second feature map from the instrument image and performing local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image, the following steps 4A to 4B may be referred to:

[0140] In step 4A, for each Transform unit, the input instrument image or the features output by the previous Transform unit are processed by the attention layer (Efficient Self-Attn) in the Transform unit, and the features output by the attention layer (Efficient Self-Attn) are extracted by the PAN-FFN network in the Transform unit. Then, Overlap Patch Merging is used to merge the overlapping image blocks into a complete image to obtain the second feature map output by the Transform unit.

[0141] Among them, the PAN network transfers semantic information from low dimensions to high dimensions again. It is based on the idea of ​​PANnet and has considerable advantages in detecting and segmenting small targets and detailed targets. The boundaries of the instrument panel scale data are unclear and the outline is narrow. It is a classic small target detection and segmentation. Therefore, combining the PAN network will greatly improve the detection and segmentation effect of the instrument panel scale.

[0142] The process of PAN-FFN network processing input features is as follows: using the third convolution kernel to perform a convolution operation on the features output by the attention layer, and using the Dw-Conv layer to perform a convolution operation on the features output by the attention layer; wherein the step size of the third convolution kernel is consistent with the step size of the Dw-Conv layer; fusing the convolution result of the third convolution kernel with the convolution result of Dw-Con; using the fourth convolution kernel to perform a convolution operation on the fused features, and using the Dw-D-Conv layer to perform a convolution operation on the convolution result of the Dw-Conv layer; wherein the step size of the fourth convolution kernel is consistent with the step size of the Dw-D-Conv layer; fusing the convolution result of the fourth convolution kernel with the convolution result of Dw-D-Conv; using the fifth convolution kernel to perform a convolution operation on the fused features, and using the sixth convolution kernel to perform a convolution operation on the convolution result of Dw-D-Conv; the convolution result of the fifth convolution kernel is concat spliced ​​with the convolution result of the sixth convolution kernel and then output.

[0143] Among them, the third convolution kernel can be a 3×3 convolution kernel, the fourth convolution kernel can be a 3×3 convolution kernel, the fifth convolution kernel can be a 1×1 convolution kernel, and the sixth convolution kernel can be a 1×1 convolution kernel. Based on this, see Figure 6 The schematic diagram of the structure of a PAN-FFN network shown in FIG. 1 includes two 3×3 convolution kernels, two 1×1 convolution kernels, a Dw-Conv layer, and a Dw-D-Conv layer.

[0144] On this basis, firstly, for each layer of feature input, a 3×3 convolution kernel is used for convolution operation, and the convolution step size is the same as the Dw-Conv step size, and then the output layer is fused with the Dw-Conv output layer, and then the fused feature map is subjected to a 3×3 convolution operation, and the convolution step size is the same as the Dw-D-Conv step size, and then the output layer is fused with the Dw-D-Conv output layer, and the fused feature map is subjected to a 1×1 convolution again, and the convolution result is concat spliced ​​with the 1×1Conv result, and finally the overall splicing result is output. The overall process is a complete PAN structure, which realizes the overall process of the PAN-FFN algorithm module.

[0145] In step 4B, the second feature map output by each Transform unit is processed locally and globally through a lightweight All-MLP decoder to generate a pointer and scale mask image corresponding to the instrument image.

[0146] (7) The post-processing process of the pointer and scale mask image includes the following steps 5A to 5C:

[0147] Step 5A, image corrosion processing is performed on the pointer and scale mask images. Specifically, since the pointer and scale are both small areas, in order to accurately calculate the position of the pointer relative to the scale, the image corrosion technology is used to process the image obtained by semantic segmentation to refine the scale line and the pointer to reduce noise interference. The morphological opening and closing operation is derived from the opening and closing principle in mathematical morphology. The expansion and corrosion operation of image processing is used to eliminate some interference information in the image and enhance the texture information of the image. The opening operation uses a template of the same size as the structural element to perform a convolution operation with the binary image, and then updates the binary image with the value obtained after the convolution, that is, first corrode and then expand. This operation can be used to remove noise points at the edge of the image and connect adjacent areas. The closing operation uses the same operation process as the opening operation, but first expands and then corrodes, which is used to fill the holes inside the image and connect adjacent areas. The opening operation can remove isolated small dots and burrs; the closing operation can fill small holes and bridge cracks, but the shape of the image remains unchanged.

[0148] Step 5B, expand the pointer and scale mask image after the image corrosion process into a rectangular image. Specifically, in order to facilitate the calculation of the reading, it is necessary to convert the annular dial in the image after corrosion into a rectangular image, and the dial image can be expanded from a circle to a rectangle using polar coordinate transformation. The center of the dial scale area obtained by semantic segmentation is used as the transformation center to perform polar coordinate transformation to transform the image into a polar coordinate system.

[0149]

[0150] Where ρ and θ are the polar radius and polar angle of the polar coordinate system, respectively, and x c ,y c is the transformation center, x0 and y0 are the horizontal and vertical coordinates of the pixel in the original coordinate system. Since the starting point of the polar coordinate expansion does not correspond to the circular dial scale position, the polar coordinates can be obtained through the starting and ending position information of the circular dial scale. After the transformation, the image is re-stitched with the corresponding position information on the rectangular dial to obtain a rectangular dial image containing only the scale area.

[0151] Step 5C, based on the relative position of the positioning pointer in the rectangular image and the dial range determined based on the rectangular image, the meter reading result corresponding to the image to be detected is determined according to the position and the dial range, such as Figure 7 A schematic diagram of a meter reading result is shown, and Figure 8 Schematic diagram of another meter reading result shown in FIG. Specifically, in the processed output image, the position of the pointer relative to the scale is located by a threshold, and then the dial type is determined according to the number of scales to obtain the dial range, and finally the reading is calculated according to the relative position of the pointer and the range through the formula:

[0152]

[0153] Where: V is the current instrument reading; n is the number of scale lines corresponding to the full scale; M is the range; h is the number of scale lines corresponding to the position of the pointer relative to the scale

[0154] In summary, the embodiments of the present invention provide Fig. 9 The overall flow chart of an industrial pointer instrument detection reading recognition method shown includes:

[0155] Step S902, obtain industrial field instrument video data, save the video data as samples by alternate frame saving method, and form a sample set. Specifically, generate samples, obtain industrial field instrument video data, save the video data by alternate frame saving method, and form a sample set.

[0156] Step S904: Sample labeling, constructing an instrument target detection dataset. Specifically, the target detection sample set is labeled, and the instrument targets in the image are labeled using the labeling software labelme. Each image can generate a corresponding xml file to form an instrument detection dataset.

[0157] Step S906: Sample annotation, constructing an instrument semantic segmentation dataset. Specifically, the semantic segmentation sample set annotation uses annotation software to annotate the instrument dial scales and pointers in the image, and each image can generate a corresponding mask image to form an instrument semantic segmentation dataset.

[0158] Step S908, the samples are expanded by operations such as rotation, flipping, brightness, scaling, translation and cropping. Specifically, the sample enhancement is to perform enhancement operations such as rotation, flipping, saturation, brightness, scaling, translation and cropping on the instrument detection data set to expand the sample data set.

[0159] Step S910, build the target detection network structure YOLOV5, filter out images with instruments in the image as sample sets for training, the algorithm proposes a top-down path and a bottom-up path in the Neck part, which is a combination of FPN+PAN structure. Among them, FPN, namely the Feature Pyramid Network, is used to process the output from multiple scales in the Backbone, and then construct high-level semantic feature information, while PAN, namely the Path Aggregation Network, uses a bottom-up structure to make up for and fuse strong positioning information, and finally outputs it for Head detection. It can enhance the extraction of feature information in the target detection process, thereby effectively improving the performance of target detection.

[0160] Step S912, build a semantic segmentation network structure Segformer, filter out complete instrument images in the image as a sample set for training, the algorithm proposes a simple, efficient, but powerful semantic segmentation framework, which combines Transformers with a lightweight multi-layer perception (MLPs) decoder. The Transformer encoder can output multi-scale features, but does not require position encoding, so there is no need to interpolate the position encoding; the MLP decoder gathers information from different layers, thereby combining local attention and global attention to present a powerful representation, which can express more comprehensive feature information.

[0161] Step S914, train the target detection network, and use the labeled instrument target detection data for training and prediction. Specifically, the target detection basic model training uses the existing natural image COCO target detection open source dataset to train the basic model; then use the instrument target detection dataset for transfer learning, use the weight parameters of the trained basic model to initialize the network, and then perform model training on the enhanced instrument dataset to speed up the convergence of the model.

[0162] Step S916, training the semantic segmentation network, using the labeled instrument semantic segmentation data for training and prediction. Specifically, for semantic segmentation basic model training, use the existing natural image ADE20K semantic segmentation open source dataset to train the basic model, then use the instrument semantic segmentation dataset for transfer learning, use the weight parameters of the trained basic model to initialize the network, and then perform model training on the enhanced instrument semantic segmentation dataset to speed up the convergence of the model and improve the accuracy of the model.

[0163] Step S918, the object detection and semantic segmentation models trained under different versions and frameworks are connected in series through onnx conversion.

[0164] Step S920, detect the instrument, perform instrument tilt correction on the detected instrument, and segment the instrument panel and pointer of the corrected instrument through the semantic model.

[0165] Step S922, the segmented dashboard image is subjected to an erosion and dilation operation to reduce image noise, and then the annular dashboard outline is expanded into a rectangular image.

[0166] Step S924, by locating the pointer relative to the scale position, and according to the dial range, the meter reading is calculated.

[0167] The industrial pointer instrument detection reading recognition method provided by the embodiment of the present invention uses multi-scale feature fusion, multi-stage model detection and model post-processing, and performs transfer learning based on the existing natural instrument image data set to realize the industrial pointer instrument detection reading recognition. The embodiment of the present invention uses the above-mentioned target detection model and semantic segmentation model to significantly improve the accuracy of industrial pointer instrument detection reading recognition, thereby improving the safe use of different instruments in industrial scenes and ensuring that industrial production can proceed stably.

[0168] Based on the above embodiments, the present invention provides an industrial pointer instrument detection reading recognition device, see Fig.10 The structure diagram of an industrial pointer instrument detection reading recognition device shown in the figure mainly includes the following parts:

[0169] The image acquisition module 1002 is used to acquire the image to be detected;

[0170] The target detection module 1004 is used to extract a multi-scale first feature map from the image to be detected by using the pre-trained target detection model, construct multi-scale high-level semantic feature information based on the multi-scale first feature map, and perform layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract the instrument image from the image to be detected;

[0171] The semantic segmentation module 1006 is used to extract a multi-scale second feature map from the instrument image through a pre-trained semantic segmentation model, and perform local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image;

[0172] The meter reading module 1008 is used to determine the meter reading result corresponding to the image to be detected based on the pointer and scale mask image.

[0173] The above-mentioned device provided by the embodiment of the present invention extracts the multi-scale first feature of the image to be detected through the target detection model, and further constructs multi-scale high-level semantic feature information, and then performs layer-by-layer feature fusion to extract the instrument image from the image to be detected, and on this basis continues to use the semantic segmentation model to extract the multi-scale second feature map from the instrument image, and generates the corresponding pointer and scale mask image through local and global attention processing, and post-processes the mask image to obtain the final instrument reading result. The embodiment of the present invention uses the above-mentioned target detection model and semantic segmentation model to significantly improve the accuracy of industrial pointer instrument detection reading recognition, thereby improving the safe use of different instruments in industrial scenes and ensuring that industrial production can proceed stably.

[0174] In one embodiment, the target detection model includes a backbone network, the backbone network includes a Focus unit, a feature extraction unit and an SPP unit, and the feature extraction units each include an alternately connected CBL sub-network and a large selection kernel sub-network; the target detection module 1004 is also used to:

[0175] Through the Focus unit, pixels are periodically extracted from the high-resolution image to be detected and reconstructed into a low-resolution image;

[0176] In the feature extraction unit, after the low-resolution image is processed by the CBL sub-network, the input features are extracted by the large selection kernel sub-network, and the features output by the large selection kernel network are processed by the next CBL sub-network. This process is repeated until the last CBL sub-network outputs the features.

[0177] Through the SPP unit, the features output by the last CBL sub-network are pooled and fused;

[0178] The features output by the large selection kernel network specified in the feature extraction unit and the output features of the SPP unit are used as the first multi-scale feature map.

[0179] In one embodiment, the target detection model further includes a feature pyramid network, which includes a plurality of large selection kernel networks and a plurality of first feature processing networks from bottom to top; the target detection module 1004 is further used to:

[0180] The first feature map is input into the large selection kernel network in the feature pyramid network, and the input features are extracted through the large selection kernel network;

[0181] In the first feature processing network, after the features output by the large selection kernel sub-network are processed by the CBL sub-network in the first feature processing network, the features output by the CBL sub-network are upsampled, and the upsampled features are fused with the first feature map having the same size as the features;

[0182] The fused features are input into the next large selection kernel network in the feature pyramid network, and the input features are extracted through the large selection kernel network;

[0183] In the next first feature processing network, the features output by the large selection kernel network are upsampled, and the upsampled features are fused with the first feature map of the same size to obtain multi-scale high-level semantic feature information.

[0184] In one embodiment, the target detection model further includes a channel aggregation network, which includes a plurality of large selection kernel networks and a plurality of second feature processing networks from top to bottom; the target detection module 1004 is further used to:

[0185] Input high-level semantic feature information into the large selection kernel network in the channel aggregation network, and extract the input features through the large selection kernel network;

[0186] In the second feature processing network, after the features output by the large selection kernel sub-network are processed by the CBL sub-network in the first feature processing network, the features output by the CBL sub-network are fused with the high-level semantic feature information consistent with its size;

[0187] Continue to input the fused features into the next large selection kernel network in the channel aggregation network, and repeat this process until the last large selection kernel network in the channel aggregation network outputs features;

[0188] The instrument image is generated based on the features output by each large selection kernel sub-network in the channel aggregation network.

[0189] In one implementation, the target detection module 1004 is further configured to:

[0190] Use the first convolution kernel to perform a convolution operation on the input features;

[0191] Using the second convolution kernel to perform a convolution operation on the convolution operation result of the first convolution kernel;

[0192] Concatenate the convolution operation result of the first convolution kernel with the convolution operation result of the second convolution kernel;

[0193] After Avgpool, convolution, and sigmoid operations, the concat result is subjected to dot multiplication with the convolution operation results of the first convolution kernel and the convolution operation results of the second convolution kernel respectively;

[0194] The dot product results are concat-concatenated, and the concat result is dot-multiplied with the input features before output.

[0195] In one embodiment, the semantic segmentation model includes a hierarchical Transformer encoder and a lightweight All-MLP decoder, the hierarchical Transformer encoder includes multiple Transform units, each Transform unit includes multiple feature extraction subunits and merging subunits, each feature extraction subunit includes an attention layer and a PAN-FFN network; the semantic segmentation module 1006 is also used to:

[0196] For each Transform unit, the input instrument image or the features output by the previous Transform unit are processed through the attention layer in the Transform unit, and the features output by the attention layer are extracted through the PAN-FFN network in the Transform unit to obtain the second feature map output by the Transform unit;

[0197] Through the lightweight All-MLP decoder, the second feature map output by each Transform unit is processed locally and globally to generate the pointer and scale mask images corresponding to the instrument image.

[0198] In one embodiment, the semantic segmentation module 1006 is further configured to:

[0199] The features output by the attention layer are convolved using the third convolution kernel, and the features output by the attention layer are convolved using the Dw-Conv layer; wherein the step size of the third convolution kernel is consistent with the step size of the Dw-Conv layer;

[0200] The convolution result of the third convolution kernel is merged with the convolution result of Dw-Con;

[0201] The fourth convolution kernel is used to perform a convolution operation on the fused features, and the Dw-D-Conv layer is used to perform a convolution operation on the convolution result of the Dw-Conv layer; wherein the step size of the fourth convolution kernel is consistent with the step size of the Dw-D-Conv layer;

[0202] The convolution result of the fourth convolution kernel is merged with the convolution result of Dw-D-Conv;

[0203] The fifth convolution kernel is used to perform a convolution operation on the fused features, and the sixth convolution kernel is used to perform a convolution operation on the convolution result of Dw-D-Conv;

[0204] The convolution result of the fifth convolution kernel is concatenated with the convolution result of the sixth convolution kernel and then output.

[0205] In one embodiment, the meter reading module 1008 is further configured to:

[0206] Perform image corrosion processing on the pointer and scale mask images;

[0207] Expand the pointer and scale mask images after image corrosion processing into a rectangular image;

[0208] Based on the position of the positioning pointer relative to the scale in the rectangular image and the dial range determined based on the rectangular image, the instrument reading result corresponding to the image to be detected is determined according to the position and the dial range.

[0209] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0210] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0211] Fig.11 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 110, a memory 111, a bus 112 and a communication interface 113, the processor 110, the communication interface 113 and the memory 111 are connected via the bus 112; the processor 110 is used to execute an executable module stored in the memory 111, such as a computer program.

[0212] The memory 111 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 113 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0213] The bus 112 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0214] Among them, the memory 111 is used to store programs, and the processor 110 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any of the embodiments of the present invention can be applied to the processor 110 or implemented by the processor 110.

[0215] The processor 110 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 110. The above processor 110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 111, and the processor 110 reads the information in the memory 111 and completes the steps of the above method in combination with its hardware.

[0216] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0217] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0218] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for detecting and identifying readings of industrial pointer instruments, characterized in that: include: Acquire the image to be detected; Extracting a multi-scale first feature map from the image to be detected by using a pre-trained target detection model, constructing multi-scale high-level semantic feature information based on the multi-scale first feature map, and performing layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract an instrument image from the image to be detected; Extracting a multi-scale second feature map from the instrument image through a pre-trained semantic segmentation model, and performing local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image; Based on the pointer and scale mask image, determining the instrument reading result corresponding to the image to be detected; The target detection model includes a backbone network, the backbone network includes a Focus unit, a feature extraction unit and an SPP unit, and the feature extraction units each include an alternately connected CBL sub-network and a large selection kernel sub-network; The step of extracting a multi-scale first feature map from the image to be detected includes: By means of the Focus unit, pixel points are periodically extracted from the high-resolution image to be detected and reconstructed into a low-resolution image; In the feature extraction unit, after the CBL sub-network processes the low-resolution image, the input feature is extracted through the large selection kernel sub-network, and the feature output by the large selection kernel sub-network is processed through the next CBL sub-network, and this process is repeated until the last CBL sub-network outputs a feature; Through the SPP unit, the features output by the last CBL sub-network are pooled and fused; Using the features output by the large selection kernel network specified in the feature extraction unit and the output features of the SPP unit as a multi-scale first feature map; The semantic segmentation model includes a hierarchical Transformer encoder and a lightweight All-MLP decoder, the hierarchical Transformer encoder includes a plurality of Transform units, each of the Transform units includes a plurality of feature extraction subunits and merging subunits, and each of the feature extraction subunits includes an attention layer and a PAN-FFN network; The step of extracting a multi-scale second feature map from the instrument image and performing local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image includes: For each Transform unit, the input instrument image or the features output by the previous Transform unit are processed by the attention layer in the Transform unit, and the features output by the attention layer are extracted by the PAN-FFN network in the Transform unit to obtain a second feature map output by the Transform unit; The lightweight All-MLP decoder performs local and global attention processing on the second feature map output by each Transform unit to generate a pointer and scale mask image corresponding to the instrument image.

2. The industrial pointer instrument detection reading recognition method according to claim 1 is characterized in that: The target detection model also includes a feature pyramid network, which includes a plurality of large selection kernel networks and a plurality of first feature processing networks from bottom to top; The step of constructing multi-scale high-level semantic feature information based on the multi-scale first feature map includes: Inputting the first feature map into the large selection kernel network in the feature pyramid network, and performing feature extraction on the input features through the large selection kernel network; In the first feature processing network, after the features output by the large selection kernel subnetwork are processed by the CBL subnetwork in the first feature processing network, the features output by the CBL subnetwork are upsampled, and the upsampled features are fused with the first feature map having the same size as the features; Inputting the fused features into the next large selection kernel network in the feature pyramid network, and extracting the input features through the large selection kernel network; In the next first feature processing network, the features output by the large selection kernel network are upsampled, and the upsampled features are fused with the first feature map of the same size to obtain multi-scale high-level semantic feature information.

3. The industrial pointer instrument detection reading recognition method according to claim 2 is characterized in that: The target detection model also includes a channel aggregation network, which includes a plurality of top-down large selection kernel networks and a plurality of second feature processing networks; The step of performing layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract the instrument image from the image to be detected includes: Inputting the high-level semantic feature information into the large selection kernel network in the channel aggregation network, and extracting the input features through the large selection kernel network; In the second feature processing network, after the features output by the large selection kernel sub-network are processed by the CBL sub-network in the first feature processing network, the features output by the CBL sub-network are fused with the high-level semantic feature information having the same size as the features output by the CBL sub-network; Continue to input the fused features into the next large selection nuclear sub-network in the channel aggregation network, and repeat this process until the last large selection nuclear sub-network in the channel aggregation network outputs features; An instrument image is generated based on features output by each of the large selection kernel sub-networks in the channel aggregation network.

4. The method for detecting and identifying readings of an industrial pointer instrument according to any one of claims 1 to 3, characterized in that: The step of extracting features from input features through the large selection kernel network includes: Use the first convolution kernel to perform a convolution operation on the input features; Using a second convolution kernel to perform a convolution operation on the convolution operation result of the first convolution kernel; Concatenate the convolution operation result of the first convolution kernel and the convolution operation result of the second convolution kernel; After Avgpool, convolution, and sigmoid operations, the concat result is subjected to dot multiplication operations with the convolution operation results of the first convolution kernel and the convolution operation results of the second convolution kernel respectively; The dot product results are concat-concatenated, and the concat result is dot-multiplied with the input features before output.

5. The industrial pointer instrument detection reading recognition method according to claim 1, characterized in that: The step of extracting features of the output of the attention layer through the PAN-FFN network in the Transform unit includes: Using a third convolution kernel to perform a convolution operation on the features output by the attention layer, and using a Dw-Conv layer to perform a convolution operation on the features output by the attention layer; wherein the step size of the third convolution kernel is consistent with the step size of the Dw-Conv layer; Fusion the convolution result of the third convolution kernel with the convolution result of the Dw-Con; Using a fourth convolution kernel to perform a convolution operation on the fused features, and using a Dw-D-Conv layer to perform a convolution operation on the convolution result of the Dw-Conv layer; wherein the step size of the fourth convolution kernel is consistent with the step size of the Dw-D-Conv layer; Fusion the convolution result of the fourth convolution kernel and the convolution result of the Dw-D-Conv; Using the fifth convolution kernel to perform a convolution operation on the fused features, and using the sixth convolution kernel to perform a convolution operation on the convolution result of the Dw-D-Conv; The convolution result of the fifth convolution kernel is concatenated with the convolution result of the sixth convolution kernel and then output.

6. The industrial pointer instrument detection reading recognition method according to claim 1, characterized in that: The step of determining the instrument reading result corresponding to the image to be detected based on the pointer and scale mask image comprises: Performing image corrosion processing on the pointer and scale mask images; Expanding the pointer and scale mask image after the image corrosion process into a rectangular image; Based on the position of the positioning pointer relative to the scale in the rectangular image, and based on the rectangular image to determine the dial range, the instrument reading result corresponding to the image to be detected is determined according to the position and the dial range.

7. An industrial pointer instrument detection reading recognition device, characterized in that: include: An image acquisition module, used for acquiring an image to be detected; The target detection module is used to extract a multi-scale first feature map from the image to be detected by using a pre-trained target detection model, construct multi-scale high-level semantic feature information based on the multi-scale first feature map, and perform layer-by-layer feature fusion processing on the multi-scale high-level semantic feature information to extract an instrument image from the image to be detected; A semantic segmentation module, configured to extract a multi-scale second feature map from the instrument image by using a pre-trained semantic segmentation model, and perform local and global attention processing on the multi-scale second feature map to generate a pointer and scale mask image corresponding to the instrument image; An instrument reading module, used to determine the instrument reading result corresponding to the image to be detected based on the pointer and scale mask image; The target detection model includes a backbone network, which includes a Focus unit, a feature extraction unit and an SPP unit, and each of the feature extraction units includes an alternately connected CBL sub-network and a large selection kernel sub-network; the target detection module is specifically used for: By means of the Focus unit, pixel points are periodically extracted from the high-resolution image to be detected and reconstructed into a low-resolution image; In the feature extraction unit, after the CBL sub-network processes the low-resolution image, the input feature is extracted through the large selection kernel sub-network, and the feature output by the large selection kernel sub-network is processed through the next CBL sub-network, and this process is repeated until the last CBL sub-network outputs a feature; Through the SPP unit, the features output by the last CBL sub-network are pooled and fused; Using the features output by the large selection kernel network specified in the feature extraction unit and the output features of the SPP unit as a multi-scale first feature map; The semantic segmentation model includes a hierarchical Transformer encoder and a lightweight All-MLP decoder, the hierarchical Transformer encoder includes multiple Transform units, each of the Transform units includes multiple feature extraction subunits and merging subunits, each of the feature extraction subunits includes an attention layer and a PAN-FFN network; the semantic segmentation module is specifically used for: For each Transform unit, the input instrument image or the features output by the previous Transform unit are processed by the attention layer in the Transform unit, and the features output by the attention layer are extracted by the PAN-FFN network in the Transform unit to obtain a second feature map output by the Transform unit; The lightweight All-MLP decoder performs local and global attention processing on the second feature map output by each Transform unit to generate a pointer and scale mask image corresponding to the instrument image.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 6.

Images