Instrument image recognition method, device, electronic device and computer-readable medium

Abstract

The embodiments of the present disclosure disclose an instrument image recognition method, apparatus, electronic device, and computer-readable medium. A specific implementation of the method includes: performing image recognition on a pre-acquired instrument image using a preset instrument image recognition model to generate recognition information; extracting an image region corresponding to a recognition frame included in the recognition information from the instrument image to obtain a captured image; performing correction processing on the captured image to generate a corrected instrument image and a corrected scale point coordinate set; performing pointer recognition on the corrected instrument image using a preset pointer recognition model to obtain pointer recognition information; and generating an instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer recognition information, and the recognition frame center coordinates and recognition frame width values ​​included in the recognition information. This implementation can improve the accuracy of generating instrument image recognition results.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to an instrument image recognition method, apparatus, electronic device, and computer-readable medium. Background Art

[0002] Instrument image recognition is a technology used to identify instrument data in instrument images. Currently, the commonly used approach for instrument image recognition is to directly use the collected instrument images to train a model, and then use the trained model to perform instrument image recognition.

[0003] However, when using the above method for instrument image recognition, the following technical problems often occur:

[0004] First, the simultaneous recognition of the pointer and dial of the instrument image causes the pointer features and the dial features to influence each other, resulting in a high degree of coupling, which reduces the accuracy of instrument image recognition;

[0005] Second, it is difficult to collect pointer status images of the entire measuring range to train the instrument image recognition model. As a result, the instrument image recognition model generated by the commonly used method has low robustness, thereby reducing the efficiency of instrument image recognition. Summary of the Invention

[0006] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure provide an instrument image recognition method, apparatus, electronic device, and computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide an instrument image recognition method, the method comprising: performing image recognition on a pre-acquired instrument image through a preset instrument image recognition model to generate recognition information, wherein the above-mentioned recognition information includes a recognition border, a recognition border center coordinate and a recognition border width value, and the above-mentioned recognition border is composed of a border coordinate group; cutting out an image area corresponding to the recognition border included in the above-mentioned recognition information from the above-mentioned instrument image to obtain a cut-off image; correcting the above-mentioned cut-off image based on preset template image information to generate a corrected instrument image and a corrected scale point coordinate set; performing pointer recognition on the above-mentioned corrected instrument image through a preset pointer recognition model to obtain pointer recognition information; generating an instrument image recognition result based on the above-mentioned corrected instrument image, the above-mentioned correction scale point coordinate set, the above-mentioned pointer recognition information and the recognition border center coordinate and the recognition border width value included in the above-mentioned recognition information, wherein the above-mentioned instrument image recognition result includes the instrument image pointer scale value.

[0009] In a second aspect, some embodiments of the present disclosure provide an instrument image recognition device, which includes: an image recognition unit, configured to perform image recognition on a pre-acquired instrument image through a preset instrument image recognition model to generate recognition information, wherein the above-mentioned recognition information includes a recognition border, the center coordinates of the recognition border, and the width value of the recognition border, and the above-mentioned recognition border is composed of a border coordinate group; an image capture unit, configured to capture an image area corresponding to the recognition border included in the above-mentioned recognition information from the above-mentioned instrument image to obtain a captured image; a correction processing unit, configured to perform correction processing on the above-mentioned capture image based on preset template image information to generate a corrected instrument image and a set of corrected scale point coordinates; a pointer recognition unit, configured to perform pointer recognition on the above-mentioned corrected instrument image through a preset pointer recognition model to obtain pointer recognition information; a generation unit, configured to generate an instrument image recognition result based on the above-mentioned corrected instrument image, the above-mentioned correction scale point coordinate set, the above-mentioned pointer recognition information, and the center coordinates of the recognition border and the width value of the recognition border included in the above-mentioned recognition information, wherein the above-mentioned instrument image recognition result includes the instrument image pointer scale value.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0012] The above-described various embodiments of the present disclosure have the following beneficial effects: The instrument image recognition methods of some embodiments of the present disclosure can improve the accuracy of instrument image recognition. Specifically, the reason for the reduced accuracy of instrument image recognition is that the pointer and dial recognition of the instrument image are performed simultaneously, causing the pointer features and the dial features to influence each other, resulting in a high degree of coupling. Based on this, the instrument image recognition methods of some embodiments of the present disclosure introduce an instrument image recognition model and a pointer recognition model. This allows the instrument image pointer recognition process and the instrument image dial recognition process to be separated to achieve a decoupling effect. Thus, the accuracy of instrument image recognition can be improved. Furthermore, by introducing preset template image information, the captured image is corrected. This makes the instrument image recognition result generated based on the corrected instrument image, the corrected scale point coordinate set, the pointer recognition information, and the recognition information including the center coordinates and width values ​​of the recognition border more accurate. Thus, the accuracy of instrument image recognition can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a schematic diagram of an application scenario of the instrument image recognition method in some embodiments of the present disclosure;

[0015] Figure 2 is a flow chart of some embodiments of the instrument image recognition method according to the present disclosure;

[0016] Figure 3 is a flow chart of other embodiments of the instrument image recognition method according to the present disclosure;

[0017] Figure 4 is a schematic structural diagram of some embodiments of the instrument image recognition device according to the present disclosure;

[0018] Figure 5 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1 It is a schematic diagram of an application scenario of the obstacle information generation method of some embodiments of the present disclosure.

[0026] exist Figure 1In an application scenario, the computing device 101 may first perform image recognition on a pre-acquired instrument image 103 using a preset instrument image recognition model 102 to generate identification information 104, wherein the identification information 104 includes an identification border 1041, identification border center coordinates 1042, and an identification border width 1043. The identification border 1041 is composed of a border coordinate group. Next, the computing device 101 may extract from the instrument image 103 an image region corresponding to the identification border 1041 included in the identification information 104, to obtain a cutout image 105. The computing device 101 may then perform correction processing on the cutout image 105 based on preset template image information 106 to generate a corrected instrument image 107 and a set of corrected scale point coordinates 108. Thereafter, the computing device 101 may perform pointer recognition on the corrected instrument image 107 using a preset pointer recognition model 109 to obtain pointer recognition information 110. Finally, the computing device 101 can generate an instrument image recognition result 111 based on the above-mentioned corrected instrument image 107, the above-mentioned corrected scale point coordinate set 108, the above-mentioned pointer identification information 110 and the identification border center coordinates 1042 and the identification border width value 1043 included in the above-mentioned identification information 104, wherein the above-mentioned instrument image recognition result 111 includes the instrument image pointer scale value 1111.

[0027] It should be noted that the computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitations are given here.

[0028] It should be understood that Figure 1 The number of computing devices in the embodiment is merely illustrative. Any number of computing devices may be provided according to implementation requirements.

[0029] Continue to refer Figure 2 , shows a process 200 of some embodiments of the instrument image recognition method according to the present disclosure. The process 200 of the instrument image recognition method includes the following steps:

[0030] Step 201 : performing image recognition on a pre-acquired instrument image using a preset instrument image recognition model to generate recognition information.

[0031] In some embodiments, the execution subject of the instrument image recognition method (such as Figure 1The computing device 101 shown) can perform image recognition on the pre-acquired instrument image through a preset instrument image recognition model to generate recognition information. The above-mentioned recognition information may include a recognition border, the coordinates of the center of the recognition border, and the width value of the recognition border. The above-mentioned recognition border may be composed of a border coordinate group. The above-mentioned preset instrument image recognition model may be: YOLO (You only look once, single target detection) model, mobilenetV2 (lightweight target detection) model, FPN (Feature Pyramid Networks, feature pyramid network), etc. The above-mentioned pre-acquired instrument image may be an image taken of a single instrument. The above-mentioned recognition border may be an circumscribed rectangular frame of the image area representing the instrument in the instrument image.

[0032] Step 202 : extracting an image region corresponding to the identification frame included in the identification information from the instrument image to obtain a captured image.

[0033] In some embodiments, the execution entity may extract an image region corresponding to the identification border included in the identification information from the instrument image to obtain a captured image. The recognition border may be used as a capture boundary for image capture. Thus, the image region enclosed by the recognition border may be determined as the captured image. Image capture can remove other image regions from the preset instrument image, thereby avoiding interference with instrument image recognition.

[0034] In some optional implementations of some embodiments, the execution entity extracts, from the instrument image, an image region corresponding to the identification frame included in the identification information to obtain a captured image, which may include the following steps:

[0035] The first step is to adjust the recognition border included in the above recognition information to obtain the adjusted border. Since the input of the subsequent pointer segmentation link is usually a square image, the center point of the recognition border can be kept unchanged. For example, the long side (the maximum value of the width and height of the recognition border) is used as the reference and enlarged by 20%. The other side length is changed accordingly, so that the adjusted border is also a square.

[0036] For example, the width of the recognition border can be 100 pixels, and the height can be 80 pixels. Then, the left and right borders can be increased by 10 pixels each, and the top and bottom borders can be increased by 20 pixels each, resulting in a 120 pixel × 120 pixel square border.

[0037] In the second step, the image area corresponding to the adjusted frame is cut out from the instrument image to obtain a cutout image. The cutout image may be the image area included in the adjusted frame.

[0038] Step 203 : Based on the preset template image information, the captured image is rectified to generate a rectified instrument image and a set of rectified scale point coordinates.

[0039] In some embodiments, the execution entity may perform correction processing on the captured image based on preset template image information to generate a corrected instrument image and a set of corrected scale point coordinates. The preset template image information may be annotation information generated after annotating the preset template image. The preset template image may be a clear template image of the instrument to be identified, taken of the instrument. The annotation information may be a rectangular box manually annotated on the instrument area in the instrument image to be identified. The corrected instrument image may be generated in the following manner:

[0040] In the first step, an image of a rectangular frame area circumscribed to the instrument area is cut out from the template image of the instrument to be identified as a cutout template image.

[0041] In the second step, the captured template image is resized to the same size as the captured image to obtain an adjusted image. Since the captured template image is resized, the border of the adjusted image can be determined as a replacement rectangular frame. The replacement rectangular frame can be composed of a set of replacement coordinate values.

[0042] The third step is to fuse the above-mentioned replacement rectangular box and the above-mentioned identification border to obtain a fused border. The fusion can be to determine the midpoint coordinate value of each replacement coordinate value in the replacement rectangular box and the corresponding border coordinate value in the above-mentioned identification border as a fused coordinate value to obtain a fused coordinate value group. The above-mentioned fused coordinate value group can be used to characterize the above-mentioned fused border. In addition, the replacement coordinate value in the above-mentioned replacement rectangular box and the closest border coordinate value in the above-mentioned identification border can be determined as a basic correspondence. Then, based on the replacement coordinate value and the border coordinate value with a basic correspondence, in the same direction on the above-mentioned replacement rectangular box and the identification border, the replacement coordinate value and the border coordinate value adjacent to the replacement coordinate value and the border coordinate value are respectively determined as a correspondence.

[0043] The fourth step is to cut out the image of the fused border area from the instrument image to obtain a corrected instrument image.

[0044] Thus, key points of the corrected instrument image can be identified to generate a corrected scale point coordinate set. Each corrected scale point coordinate in the corrected scale point coordinate set can be used to represent a scale value on the instrument in the instrument image.

[0045] Step 204 : performing pointer recognition on the corrected instrument image using a preset pointer recognition model to obtain pointer recognition information.

[0046] In some embodiments, the execution entity may perform pointer recognition on the corrected instrument image through a preset pointer recognition model to obtain pointer recognition information. The preset pointer recognition model may be: BiSeNet (real-time semantic segmentation algorithm), DeepLab (Semantic Image Segmentation with DeepConvolutional Nets, Atrous Convolution and Fully Connected CRFs, semantic image segmentation using deep convolutional networks, Atrous convolution and fully connected CRF) algorithm, etc. The pointer recognition information may include a semantic segmentation image and a corresponding bounding box of a target area in the semantic segmentation image. The bounding box of the target area may also be composed of a target area coordinate group. The bounding box of the target area may be used to characterize the pointer area in the instrument image.

[0047] Step 205 : Generate an instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer identification information, and the identification frame center coordinates and the identification frame width value included in the identification information.

[0048] In some embodiments, the execution entity may generate an instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer identification information, and the identification frame center coordinates and identification frame width values ​​included in the identification information. The instrument image recognition result may be generated by the following steps:

[0049] In the first step, the rectified instrument image is subjected to image recognition using the instrument image recognition model to generate rectified recognition information, wherein the rectified recognition information may include the coordinates of the rectified recognition center.

[0050] In the second step, the midpoint of the line between the center coordinates of the recognition frame included in the recognition information and the corrected recognition center coordinates is determined as the target center point to obtain the target center point coordinates.

[0051] In the third step, the line connecting the target center point coordinates and the farthest target area coordinates in the target area coordinate group constituting the frame of the target area in the pointer identification information is determined as the pointer center line.

[0052] In the fourth step, a ray formed by the target center point coordinates and the midline of the pointer is used to define the coordinates of the correction scale point closest to the ray as the first target correction scale point coordinates. The coordinates of the correction scale point closest to the ray in the correction scale point coordinates set, excluding the first target correction scale point coordinates, are defined as the second target correction scale point coordinates.

[0053] In the fifth step, the line connecting the coordinates of the first target correction scale point and the coordinates of the target center point is determined as the first standard scale line, and the line connecting the coordinates of the second target correction scale point and the coordinates of the target center point is determined as the second standard scale line.

[0054] Step 6: Determine the angle between the first standard scale line and the center line of the pointer to obtain a first angle value. Determine the angle between the second standard scale line and the center line of the pointer to obtain a second angle value.

[0055] Step 7: Determine the scale value represented by the coordinates of the first target correction scale point as the first scale value. Determine the scale value represented by the coordinates of the second target correction scale point as the second scale value.

[0056] Step 8: Generate the instrument image recognition result using the following formula:

[0057]

[0058] Where K represents the instrument image recognition result. θ1 represents the first angle value. θ2 represents the second angle value. v1 represents the first scale value. v2 represents the second scale value.

[0059] In some optional implementations of some embodiments, the instrument image recognition model is trained and generated in the following manner:

[0060] The first step is to obtain an instrument image set and a natural image set. The execution entity may obtain the instrument image set and the natural image set via a wired or wireless method. The instrument images in the instrument image set may be pre-captured images of a single instrument. The natural images in the natural image set may be captured images of any natural scene.

[0061] The second step is to fuse each instrument image in the instrument image set with each natural image in the natural image set to generate a set of images to be processed. The fusion process can be performed by fusing a single instrument image onto any natural image in the natural image set to generate the image to be processed. This improves the ability of the generated instrument image recognition model to recognize instruments in different backgrounds, thereby enhancing the robustness of the instrument image recognition model.

[0062] The third step is to perform sample processing on each of the images in the image set to be processed to generate a sample set. The sample processing may include image annotation of the images in the image set to generate the sample set. The image annotation may include a bounding rectangle that marks the instrument image region in each image in the image set. Each sample in the sample set may include the annotated image to be processed and a sample label.

[0063] The fourth step is to train the initial instrument image recognition model based on the sample set to generate the instrument image recognition model. The labeled image to be processed included in each sample in the sample set can be input into the initial instrument image recognition model for model training. Thus, the instrument image recognition model can be generated. In some optional implementations of some embodiments, the pointer recognition model is trained and generated by:

[0064] In the first step, a dial region is captured from each instrument image in the instrument image set to obtain a dial region image set. The capture step may include capturing the image region within the marked circumscribed rectangular frame from each instrument image in the instrument image set to generate a dial region image, thereby obtaining the dial region image set.

[0065] The second step is to perform pointer annotation on each dial area image in the dial area image set to obtain a pointer annotation image set. The pointer annotation can indicate the area in the dial area image where the pointer is located. For example, the image area where the pointer is located can be approximately marked as a circumscribed quadrilateral.

[0066] The third step is to perform image enhancement on each of the pointer-annotated images in the aforementioned pointer-annotated image set to generate an enhanced image set. Enhancement can involve enriching the pointer's orientation and shape through operations such as projection transformation and rotation. This improves the generated instrument image recognition model's ability to recognize pointers at different angles, thereby enhancing the robustness of the pointer recognition model.

[0067] The fourth step is to train the initial pointer recognition model using the enhanced images to generate the pointer recognition model. Each enhanced image in the enhanced image set can be input into the pointer recognition model for model training. Thus, the pointer recognition model can be generated.

[0068] The above-described various embodiments of the present disclosure have the following beneficial effects: The instrument image recognition methods of some embodiments of the present disclosure can improve the accuracy of instrument image recognition. Specifically, the reason for the reduced accuracy of instrument image recognition is that the pointer and dial recognition of the instrument image are performed simultaneously, causing the pointer features and the dial features to influence each other, resulting in a high degree of coupling. Based on this, the instrument image recognition methods of some embodiments of the present disclosure introduce an instrument image recognition model and a pointer recognition model. This allows the instrument image pointer recognition process and the instrument image dial recognition process to be separated to achieve a decoupling effect. Thus, the accuracy of instrument image recognition can be improved. Furthermore, by introducing preset template image information, the captured image is corrected. This makes the instrument image recognition result generated based on the corrected instrument image, the corrected scale point coordinate set, the pointer recognition information, and the recognition information including the center coordinates and width values ​​of the recognition border more accurate. Thus, the accuracy of instrument image recognition can be further improved.

[0069] Further references Figure 3 , which shows a process 300 of another embodiment of the instrument image recognition method. The process 300 of the instrument image recognition method includes the following steps:

[0070] Step 301: Perform semantic extraction processing on the instrument image through a semantic extraction network to generate a first semantic feature, a second semantic feature, and a third semantic feature.

[0071] In some embodiments, the execution subject of the instrument image recognition method (such as Figure 1 The computing device 101 shown can perform semantic extraction processing on the instrument image using the semantic extraction network to generate a first semantic feature, a second semantic feature, and a third semantic feature. The instrument image recognition model can include a semantic extraction network, a feature sampling network, and an instrument image recognition network. Furthermore, the semantic extraction network can include a first semantic convolution module, a second semantic convolution module, and a feature mapping module. Thus, the first semantic feature can be generated by the following steps:

[0072] In the first step, the instrument image is input into the first semantic convolution module of the semantic extraction network to generate a first semantic feature. The first semantic convolution module may include a first convolution layer, a first batch normalization layer, and a first activation layer. The convolution operation of the first convolution layer increases the number of channels in the input feature.

[0073] As an example, the convolution kernel size of the first convolutional layer may be 1×1.

[0074] In the second step, the first semantic feature is input into a second semantic convolution module included in the semantic extraction network to generate a second semantic feature. The second semantic convolution module may include a second convolution layer, a second batch normalization layer, and a second activation layer. Furthermore, the second convolution layer may be a separable convolution. Through the convolution operation of the second convolution layer, the spatial features of the first semantic feature can be learned channel by channel, reducing computational complexity.

[0075] As an example, the convolution kernel size of the second convolutional layer may be 3×3.

[0076] In the third step, the second semantic feature is input into the feature mapping module included in the semantic extraction network to generate a third semantic feature. The feature mapping module may include a third convolutional layer and a batch normalization layer. This reduces the number of feature channels, making the number of channels in the output tensor equal to the number of channels in the input tensor. Finally, a residual connection can be used, i.e., a residual structure can be introduced to mitigate the gradient diffusion and degradation problems of deep networks. Consequently, the recognition accuracy of the instrument image model can be improved.

[0077] Step 302 : Perform feature sampling processing on the first semantic feature, the second semantic feature, and the third semantic feature through a feature sampling network to generate a first sampling feature, a second sampling feature, and a third sampling feature.

[0078] In some embodiments, the execution subject may perform feature sampling processing on the first semantic feature, the second semantic feature and the third semantic feature through the feature sampling network to generate a first sampling feature, a second sampling feature and a third sampling feature. First, the first semantic feature, the second semantic feature and the third semantic feature may be upsampled respectively to obtain a first upsampled feature, a second upsampled feature and a third upsampled feature. Then, a standard convolution combination may be used to perform a smoothing operation on the number of channels, and the first upsampled feature, the second upsampled feature and the third upsampled feature may be merged into a fused feature. Finally, the fused feature may be feature extracted through a convolution combination to obtain a first sampling feature, a second sampling feature and a third sampling feature. The standard convolution combination may have the same structure as the first semantic convolution module.

[0079] As an example, the number of channels of the first sampling feature may be 64, the number of channels of the second sampling feature may be 128, and the number of channels of the third sampling feature may be 256.

[0080] Step 303 : Perform feature mapping processing on the first sampling feature, the second sampling feature, and the third sampling feature through an instrument image recognition network to generate the above-mentioned recognition information.

[0081] In some embodiments, the execution entity may perform feature mapping processing on the first sampling feature, the second sampling feature, and the third sampling feature through the instrument image recognition network to generate the identification information. The instrument image recognition network may include a first header network, a second header network, and a third header network. The identification information may be generated by the following steps:

[0082] In the first step, the first head network performs feature recognition on the first sampled features to generate first identification information. This first identification information may include at least one identification box, a corresponding confidence score for each identification box, the center coordinates of the identification box, and the width of the identification box. The identification box may be composed of a set of identification coordinates and is used to represent the image of the recognized instrument area.

[0083] In the second step, a second head network is used to perform feature recognition on the second sampled features to generate second recognition information. The second recognition information may include at least one recognition box and a confidence level corresponding to each recognition box.

[0084] In the third step, a third head network is used to perform feature recognition on the third sampled features to generate third identification information, wherein the third identification information may include at least one identification box and a confidence level corresponding to each identification box.

[0085] In the fourth step, a recognition frame having a confidence level greater than a preset threshold is selected from the first recognition information, the second recognition information, and the third recognition information as a target recognition frame to obtain a target recognition frame group.

[0086] In the fifth step, the optimal target recognition frame is selected from the above target recognition frame group as the recognition frame through the non-maximum suppression algorithm, and the recognition frame center coordinates and recognition frame width values ​​corresponding to the above recognition frame are determined as recognition information.

[0087] Step 304 : extracting an image region corresponding to the identification frame included in the identification information from the instrument image to obtain a captured image.

[0088] In some embodiments, the specific implementation of step 304 and the resulting technical effects can be referred to Figure 2 The corresponding step 202 in the embodiments will not be described in detail here.

[0089] Step 305 : Based on the preset template image information, the captured image is rectified to generate a rectified instrument image and a set of rectified scale point coordinates.

[0090] In some embodiments, the execution entity may perform correction processing on the captured image based on preset template image information to generate a corrected instrument image and a set of corrected scale point coordinates. The template image information may include a template image and a set of key points that annotate a target area in the template image. The target area may be the dial of an instrument in the template image. The corrected instrument image and the set of corrected scale point coordinates may be generated by the following steps:

[0091] In the first step, feature point detection is performed on the above-mentioned intercepted image to obtain a set of detected feature points. AKAZE (Accelerated-KAZE, a local feature matching algorithm) can be used to perform feature point detection on the above-mentioned intercepted image to obtain a set of detected feature points.

[0092] In the second step, each detection feature point in the detection feature point set is matched with each key point in the key point set to generate a set of matched feature points. The matching process may include clustering each detection feature point in the detection feature point set and each key point in the key point set using a k-means clustering algorithm to generate a set of clustered feature point groups. Then, clustered feature points in the clustered feature point group whose distance from the cluster center is greater than a preset distance threshold may be removed to generate a post-removal clustered feature point set, resulting in a post-removal clustered feature point set. Finally, the post-removal clustered feature point set may be determined as the set of matched feature points.

[0093] The third step is to generate a homography matrix based on the template image. The template image can be sampled using a random sampling consensus algorithm to generate the homography matrix.

[0094] The fourth step is to use the homography matrix to correct the captured image to generate a corrected instrument image and a set of corrected scale point coordinates. The homography matrix can be used to perform a perspective transformation on the captured image to obtain the corrected instrument image. Due to the perspective transformation of the captured image, the positions of the feature points in the captured image also change. Therefore, the homography matrix can also be used to perform a coordinate transformation on the matching feature points in the set of matching feature points to obtain a set of corrected scale point coordinates. This correction process can eliminate the impact of image feature errors caused by dial tilting, rotation, and other operations on subsequent steps.

[0095] Step 306 : Perform pointer recognition on the corrected instrument image using a preset pointer recognition model to obtain pointer recognition information.

[0096] In some embodiments, the execution entity may perform pointer recognition on the corrected instrument image using a preset pointer recognition model to obtain pointer recognition information. Specifically, the backbone network of the lightweight semantic segmentation network (Bilateral Network) may be replaced with the semantic extraction network. Thus, the network structure of the pointer recognition model may be obtained. The pointer recognition information may include a post-recognition feature map. Pixels in the pointer area of ​​the post-recognition feature map may be marked as 1, while pixels in other areas may be marked as 0.

[0097] Step 307 : Generate an instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer identification information, and the identification frame center coordinates and the identification frame width value included in the identification information.

[0098] In some embodiments, the execution entity may generate an instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer identification information, and the identification frame center coordinates and identification frame width values ​​included in the identification information. The instrument image recognition result may include the instrument image pointer scale values. The instrument image recognition result may be generated by the following steps:

[0099] In the first step, arc fitting is performed on the coordinates of each correction scale point in the correction scale point coordinate set to generate an arc equation and the coordinates of the arc center point. The arc fitting can be performed on the coordinates of each correction scale point in the correction scale point coordinate set to generate the arc equation and the coordinates of the arc center point using a curve fitting method.

[0100] In the second step, based on the pointer identification information, the rectified instrument image is binarized to obtain a binary instrument image. In the identified feature map, pixels in the area where the pointer is located are marked as 1, while pixels in other areas are marked as 0. Therefore, the binarized instrument image obtained by binarizing the rectified instrument image clearly identifies the image area representing the pointer.

[0101] The third step is to generate a pointer fitting line equation based on the binarized instrument image, the coordinates of the arc center point, and the coordinates of the identification frame center and the width of the identification frame included in the recognition information. If the width of the identification frame is greater than a preset width threshold, an edge detection algorithm can be used to determine the minimum circumscribed triangle of the pointer region in the binarized instrument image. Finally, the line connecting the vertices of the circumscribed triangle and the center coordinates of the identification frame can be determined as the pointer fitting line equation. If the width of the identification frame is less than or equal to the preset width threshold, the pointer fitting line equation can be generated using the above formula and its related content, thereby generating the instrument image recognition result.

[0102] The fourth step is to generate the instrument image recognition result based on the above-mentioned pointer fitting straight line equation, the above-mentioned arc equation and the above-mentioned correction scale point coordinate set. Among them, first of all, the intersection coordinates of the above-mentioned pointer fitting straight line equation and the above-mentioned arc equation can be determined. Then, the correction scale point coordinates in the above-mentioned correction scale point coordinate set that are closest to the above-mentioned intersection coordinates can be determined as the first intersection scale coordinates. Afterwards, the correction scale point coordinates in the above-mentioned correction scale point coordinate set that are closest to the above-mentioned intersection coordinates except the above-mentioned first intersection scale coordinates can be determined as the second intersection scale coordinates. Then, the scale value corresponding to the above-mentioned first intersection scale coordinates can be the first intersection scale value. The scale value corresponding to the above-mentioned second intersection scale coordinates can be the second intersection scale value. Then, the arc length between the above-mentioned intersection coordinates and the above-mentioned first intersection scale coordinates can be determined to obtain the first intersection arc length. Determine the arc length between the above-mentioned intersection coordinates and the above-mentioned second intersection scale coordinates to obtain the second intersection arc length. Finally, the instrument image recognition result can be generated by the following formula:

[0103]

[0104] Where K represents the instrument image recognition result. L1 represents the arc length of the first intersection. L2 represents the arc length of the second intersection. a1 represents the scale value of the first intersection. a2 represents the scale value of the second intersection.

[0105] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 The process 300 of the instrument image recognition method in some corresponding embodiments embodies the steps of generating recognition information, corrected instrument images, corrected scale point coordinate sets, and instrument image recognition results. By introducing the instrument image recognition model, the pointer recognition model, and the preset template image information, a small amount of template images can be used to perform model training on the instrument image recognition model and the pointer recognition model as well as instrument image recognition. Thus, the collection of pointer status pictures of the entire range is avoided. Also, because of the introduction of the pointer recognition model and the related content for generating instrument image recognition results, the high-quality requirements for the captured natural scene images can be avoided, and the recognition ability of instrument images in scenes with dial stains, reflections, shadows, etc. is improved. Thus, the robustness of the instrument image recognition model is improved. Furthermore, the efficiency of instrument image recognition is improved. In addition, Figure 2 and Figure 3 The neural network algorithms involved in one or more of the corresponding embodiments are based on separable convolution and inverse residual modules designed for mobile computing power, with lightweight structural improvements. As a result, they can be used in low-computing portable mobile devices such as smart glasses.

[0106] Further references Figure 4As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an instrument image recognition device. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0107] like Figure 4 As shown, the instrument image recognition device 400 of some embodiments includes: an image recognition unit 401, an image interception unit 402, a correction processing unit 403, a pointer recognition unit 404 and a generation unit 405. Among them, the image recognition unit 401 is configured to perform image recognition on the pre-acquired instrument image through a preset instrument image recognition model to generate recognition information, wherein the above-mentioned recognition information includes a recognition frame, the center coordinates of the recognition frame, and the width value of the recognition frame, and the above-mentioned recognition frame is composed of a frame coordinate group; the image capture unit 402 is configured to capture an image area corresponding to the recognition frame included in the above-mentioned recognition information from the above-mentioned instrument image to obtain a captured image; the correction processing unit 403 is configured to perform correction processing on the above-mentioned capture image based on preset template image information to generate a corrected instrument image and a set of corrected scale point coordinates; the pointer recognition unit 404 is configured to perform pointer recognition on the above-mentioned corrected instrument image through a preset pointer recognition model to obtain pointer recognition information; the generation unit 405 is configured to generate an instrument image recognition result based on the above-mentioned corrected instrument image, the above-mentioned corrected scale point coordinate set, the above-mentioned pointer recognition information, and the center coordinates of the recognition frame and the width value of the recognition frame included in the above-mentioned recognition information, wherein the above-mentioned instrument image recognition result includes the instrument image pointer scale value.

[0108] It is understood that the units described in the device 400 are similar to those in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 400 and the units included therein, and will not be repeated here.

[0109] Reference below Figure 5 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of the computing device 101)500. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0111] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0112] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0113] It should be noted that in some embodiments of the present disclosure, the computer-readable medium mentioned above may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0114] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0115] The computer-readable medium may be included in the apparatus or may exist independently and not incorporated into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: perform image recognition on a pre-acquired instrument image using a preset instrument image recognition model to generate recognition information, wherein the recognition information includes a recognition frame, the center coordinates of the recognition frame, and a width value of the recognition frame, and the recognition frame is composed of a frame coordinate group; extract an image region corresponding to the recognition frame included in the recognition information from the instrument image to obtain a captured image; perform correction processing on the captured image based on preset template image information to generate a corrected instrument image and a set of corrected scale point coordinates; perform pointer recognition on the corrected instrument image using a preset pointer recognition model to obtain pointer recognition information; and generate an instrument image recognition result based on the corrected instrument image, the set of corrected scale point coordinates, the pointer recognition information, and the center coordinates of the recognition frame and the width value of the recognition frame included in the recognition information, wherein the instrument image recognition result includes the scale value of the instrument image pointer.

[0116] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The units described in some embodiments of the present disclosure may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor including an image recognition unit, an image capture unit, a correction processing unit, a pointer recognition unit, and a generation unit. The names of these units do not, in some cases, limit the units themselves. For example, the generation unit may also be described as a "unit that generates instrument image recognition results."

[0119] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0120] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for recognizing an instrument image, comprising: Performing image recognition on the pre-acquired instrument image using a preset instrument image recognition model to generate recognition information, wherein the recognition information includes a recognition frame, the coordinates of the center of the recognition frame, and a width value of the recognition frame, wherein the recognition frame is composed of a frame coordinate group; Cutting out an image area corresponding to the identification frame included in the identification information from the instrument image to obtain a cut-out image; Based on preset template image information, the captured image is corrected to generate a corrected instrument image and a set of corrected scale point coordinates, wherein the preset template image is a clear template image of the instrument to be identified that is taken of the instrument; Performing pointer recognition on the rectified instrument image using a preset pointer recognition model to obtain pointer recognition information, wherein the pointer recognition information includes a semantic segmentation image and a bounding box of a target area in the corresponding semantic segmentation image, wherein the bounding box of the target area is used to represent the pointer area in the instrument image, and the pointer area is approximately marked as a circumscribed quadrilateral; Generate an instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer identification information, and the identification frame center coordinates and identification frame width values ​​included in the identification information, wherein the instrument image recognition result includes the instrument image pointer scale value; The pointer recognition model is trained and generated in the following way: Performing dial region interception on each instrument image in the instrument image set to obtain a dial region image set; Performing pointer annotation on each dial area image in the dial area image set to obtain a pointer annotation image set, wherein pointer annotation refers to annotating the area where the pointer is located in the dial area image; performing image enhancement on each pointer-annotated image in the pointer-annotated image set to obtain an enhanced image set; Using the enhanced image to train an initial pointer recognition model to generate the pointer recognition model; The rectified instrument image is generated by: Cutting out an image of a rectangular frame area circumscribed to the instrument area from the to-be-identified instrument template image as a cut-out template image; Adjusting the captured template image to the same size as the captured image to obtain an adjusted image, and determining the border of the adjusted image as a replacement rectangular frame; Merging the replacement rectangular frame and the identification border to obtain a fused border; The image of the fused border area is cut out from the instrument image to obtain a corrected instrument image.

2. The method according to claim 1, wherein The instrument image recognition model includes: a semantic extraction network, a feature sampling network and an instrument image recognition network; and The performing image recognition on the pre-acquired instrument image by using a preset instrument image recognition model to generate recognition information includes: Performing semantic extraction processing on the instrument image through the semantic extraction network to generate a first semantic feature, a second semantic feature, and a third semantic feature; Performing feature sampling processing on the first semantic feature, the second semantic feature, and the third semantic feature through the feature sampling network to generate a first sampling feature, a second sampling feature, and a third sampling feature; The first sampling feature, the second sampling feature, and the third sampling feature are subjected to feature mapping processing through the instrument image recognition network to generate the recognition information.

3. The method according to claim 1, wherein The step of cutting out an image area corresponding to the identification frame included in the identification information from the instrument image to obtain a cut-out image includes: Adjusting the identification frame included in the identification information to obtain an adjusted frame; An image area corresponding to the adjusted frame is cut out from the instrument image to obtain a cutout image.

4. The method according to claim 1, wherein The template image information includes a template image and a set of key points annotated for a target area in the template image; as well as The correction processing of the captured image based on the preset template image information to generate a corrected instrument image and a corrected scale point coordinate set includes: Performing feature point detection on the captured image to obtain a set of detected feature points; Matching each detection feature point in the detection feature point set with each key point in the key point set to generate a matching feature point set; Based on the template image, generating a homography matrix; The captured image is rectified using the homography matrix to generate a rectified instrument image and a rectified scale point coordinate set.

5. The method according to claim 1, wherein The generating of the instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer identification information, and the identification frame center coordinates and the identification frame width value included in the identification information includes: Performing arc fitting on the coordinates of each correction scale point in the correction scale point coordinate set to generate an arc equation and the coordinates of the arc center point; Based on the pointer identification information, binarization processing is performed on the corrected instrument image to obtain a binary instrument image; Generate a pointer fitting line equation based on the binary instrument image, the arc center point coordinates, and the identification frame center coordinates and identification frame width values ​​included in the identification information; An instrument image recognition result is generated based on the pointer fitting straight line equation, the arc equation and the correction scale point coordinate set.

6. The method according to claim 1, wherein The instrument image recognition model is trained and generated in the following way: Obtain instrument image sets and natural image sets; performing fusion processing on each instrument image in the instrument image set and each natural image in the natural image set to generate an image set to be processed; Performing sample processing on each image to be processed in the image set to be processed to generate a sample set; An initial instrument image recognition model is trained based on the sample set to generate the instrument image recognition model.

7. An instrument image recognition device, comprising: an image recognition unit configured to perform image recognition on a pre-acquired instrument image using a preset instrument image recognition model to generate recognition information, wherein the recognition information includes a recognition frame, a center coordinate of the recognition frame, and a width value of the recognition frame, wherein the recognition frame is composed of a frame coordinate group; an image interception unit configured to intercept an image region corresponding to the identification frame included in the identification information from the instrument image to obtain a intercepted image; a correction processing unit configured to perform correction processing on the intercepted image based on preset template image information to generate a corrected instrument image and a set of corrected scale point coordinates, wherein the preset template image is a clear template image of the instrument to be identified that is photographed; a pointer recognition unit configured to perform pointer recognition on the rectified instrument image using a preset pointer recognition model to obtain pointer recognition information, wherein the pointer recognition information includes a semantic segmentation image and a bounding box of a target area in the corresponding semantic segmentation image, wherein the bounding box of the target area is used to represent the pointer area in the instrument image, and the pointer area is approximately marked as a circumscribed quadrilateral; a generating unit configured to generate an instrument image recognition result based on the corrected instrument image, the corrected scale point coordinate set, the pointer identification information, and the identification frame center coordinates and identification frame width values ​​included in the identification information, wherein the instrument image recognition result includes the instrument image pointer scale value; The pointer recognition model is trained and generated in the following way: Performing dial region interception on each instrument image in the instrument image set to obtain a dial region image set; Performing pointer annotation on each dial area image in the dial area image set to obtain a pointer annotation image set, wherein pointer annotation refers to annotating the area where the pointer is located in the dial area image; performing image enhancement on each pointer-annotated image in the pointer-annotated image set to obtain an enhanced image set; Using the enhanced image to train an initial pointer recognition model to generate the pointer recognition model; The rectified instrument image is generated by: Cutting out an image of a rectangular frame area circumscribed to the instrument area from the to-be-identified instrument template image as a cut-out template image; Adjusting the captured template image to the same size as the captured image to obtain an adjusted image, and determining the border of the adjusted image as a replacement rectangular frame; Merging the replacement rectangular frame and the identification border to obtain a fused border; The image of the fused border area is cut out from the instrument image to obtain a corrected instrument image.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images