Instrument identification method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a kind of instrument identification method, comprising: determining the embedded feature of pixel in target instrument image, and the position information of the pixel is encoded to obtain position encoding feature;The superposition feature obtained after the position encoding feature and the embedded feature are superimposed is input into the encoder of target model;Wherein, the input of the target model includes the label, and the output of the target model includes the coordinate of the key point in sample instrument image.According to the present disclosure, the target model obtained by training can be used to process instrument image, and output the coordinate of key point in target instrument image in target instrument image, which can reduce manual operation, improve efficiency on the one hand, and can avoid the possible misoperation in manual operation process on the other hand, which is beneficial to improve accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of display technology, and more particularly to instrument identification methods, instrument identification devices, electronic devices, and computer-readable storage media. Background Technology

[0002] Currently, instrument readings are generally performed manually. In some cases, the reader needs to physically walk to the instrument's location and observe the pointer's position on the dial to take the reading. Although in some situations an image of the instrument can be captured first, it is still necessary to manually check the image to determine the pointer's position on the dial before taking the reading.

[0003] Moreover, the readings need to be recorded after taking them, and both reading and recording operations need to be done manually, which is inefficient and inaccurate. Summary of the Invention

[0004] This disclosure provides instrument identification methods, instrument identification devices, electronic devices, and computer-readable storage media to address the shortcomings of related technologies.

[0005] According to a first aspect of the present disclosure, an instrument identification method is proposed, comprising:

[0006] The embedded features of pixels in the target instrument image are determined, and the position information of the pixels is encoded to obtain position encoded features. The target instrument image contains multiple key points related to the instrument pointer as labels.

[0007] The superimposed feature obtained by superimposing the positional encoding feature and the embedded feature is input into the encoder of the target model;

[0008] The input of the target model includes the label, and the output of the target model includes the coordinates of the key points in the sample instrument image;

[0009] The encoder includes a multi-head self-attention layer, and the target model is used to encode the superimposed features to obtain encoded features, and to determine the coordinates of the key points in the target instrument image based on the encoded features.

[0010] Optionally, before inputting the superimposed feature obtained by superimposing the location-encoded feature and the embedded feature into the target model, the method further includes:

[0011] The embedded features of pixels in the sample instrument image are determined, and the sample position information of the pixels is encoded to obtain position encoding features. The sample instrument image contains multiple sample key points related to the instrument pointer as sample labels.

[0012] The sample superposition feature obtained by superimposing the sample location encoding feature and the sample embedded feature is input into the encoder to obtain the sample encoding feature;

[0013] The training sample set is determined based on sample coding features obtained from multiple sample instrument images;

[0014] The initial model is trained based on the sample encoding features in the training sample set to obtain the target model, wherein the sample model includes the encoder, the input of the sample model includes the label, and the output of the sample model includes at least the coordinates of the sample key points in the sample instrument image.

[0015] Optionally, the key points include at least one of the following: the starting position of the dial reading, the ending position of the dial reading, the midpoint position of the dial reading, the starting position of the instrument pointer, and the ending position of the instrument pointer.

[0016] Optionally, determining the embedded features of pixels in the target instrument image and encoding the position information of the pixels to obtain position-coded features includes:

[0017] Divide the target instrument image into multiple blocks;

[0018] The embedded features of pixels in each image are determined, and the position information of the pixels is encoded to obtain position-coded features.

[0019] Optionally, the encoder includes multiple sequentially connected sub-encoders, and the target model further includes a feature pyramid;

[0020] Each sub-encoder outputs an encoding result based on the input encoding features and inputs the encoding result into the feature pyramid to obtain fused features. The size information corresponding to the encoding result output by each sub-encoder is different, and the size information corresponding to the encoding features input to each sub-encoder is different.

[0021] Optionally, the output of the target model may further include at least one of the following: the type of the target instrument image; the coordinates of at least two diagonal points of the instrument circumscribed quadrilateral in the target instrument image.

[0022] Optionally, each of the sub-encoders is connected to a linear layer, which is used to reduce the dimensionality of the encoding result output by the encoder and input the dimensionality-reduced encoding result into the next sub-encoder.

[0023] Optionally, the target model includes a first target sub-model, a second target sub-model, and a third target sub-model;

[0024] The input of the first target sub-model includes the fusion features, and the output includes the type of the target instrument image;

[0025] The input of the second target sub-model includes the fused features, and the output includes the coordinates of at least two diagonal points of the circumscribed quadrilateral of the instrument in the target instrument image;

[0026] The input of the third target sub-model includes the fused features, and the output includes the coordinates of the key points in the target instrument image.

[0027] According to a second aspect of the present disclosure, an instrument identification device is provided, comprising one or more processors configured to perform:

[0028] The embedded features of pixels in the target instrument image are determined, and the position information of the pixels is encoded to obtain position encoded features. The target instrument image contains multiple key points related to the instrument pointer as labels.

[0029] The superimposed feature obtained by superimposing the positional encoding feature and the embedded feature is input into the encoder of the target model;

[0030] The input of the target model includes the label, and the output of the target model includes the coordinates of the key points in the sample instrument image;

[0031] The encoder includes a multi-head self-attention layer, and the target model is used to encode the superimposed features to obtain encoded features, and to determine the coordinates of the key points in the target instrument image based on the encoded features.

[0032] Optionally, the processor is also configured to execute:

[0033] The embedded features of pixels in the sample instrument image are determined, and the sample position information of the pixels is encoded to obtain position encoding features. The sample instrument image contains multiple sample key points related to the instrument pointer as sample labels.

[0034] The sample superposition feature obtained by superimposing the sample location encoding feature and the sample embedded feature is input into the encoder to obtain the sample encoding feature;

[0035] The training sample set is determined based on sample coding features obtained from multiple sample instrument images;

[0036] The initial model is trained based on the sample encoding features in the training sample set to obtain the target model, wherein the sample model includes the encoder, the input of the sample model includes the label, and the output of the sample model includes at least the coordinates of the sample key points in the sample instrument image.

[0037] Optionally, the key points include at least one of the following: the starting position of the dial reading, the ending position of the dial reading, the midpoint position of the dial reading, the starting position of the instrument pointer, and the ending position of the instrument pointer.

[0038] Optionally, the processor is configured to execute:

[0039] Divide the target instrument image into multiple blocks;

[0040] The embedded features of pixels in each image are determined, and the position information of the pixels is encoded to obtain position-coded features.

[0041] Optionally, the encoder includes multiple sequentially connected sub-encoders, and the target model further includes a feature pyramid;

[0042] Each sub-encoder outputs an encoding result based on the input encoding features and inputs the encoding result into the feature pyramid to obtain fused features. The size information corresponding to the encoding result output by each sub-encoder is different, and the size information corresponding to the encoding features input to each sub-encoder is different.

[0043] Optionally, each of the sub-encoders is connected to a linear layer, which is used to reduce the dimensionality of the encoding result output by the encoder and input the dimensionality-reduced encoding result into the next sub-encoder.

[0044] Optionally, the output of the target model may further include at least one of the following: the type of the target instrument image; the coordinates of at least two diagonal points of the instrument circumscribed quadrilateral in the target instrument image.

[0045] Optionally, the target model includes a first target sub-model, a second target sub-model, and a third target sub-model;

[0046] The input of the first target sub-model includes the fusion features, and the output includes the type of the target instrument image;

[0047] The input of the second target sub-model includes the fused features, and the output includes the coordinates of at least two diagonal points of the circumscribed quadrilateral of the instrument in the target instrument image;

[0048] The input of the third target sub-model includes the fused features, and the output includes the coordinates of the key points in the target instrument image.

[0049] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing a computer program; wherein, when the computer program is executed by the processor, it implements the instrument identification method described in any of the above embodiments.

[0050] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided for storing a computer program that, when executed by a processor, implements the steps of the instrument identification method described in any of the above embodiments.

[0051] According to embodiments of this disclosure, the instrument image can be processed by the trained target model to output the coordinates of key points in the target instrument image. Then, the pointer reading in the target instrument image can be determined based on the obtained coordinates, thereby realizing the automatic determination of the pointer reading in the instrument based on the instrument image. On the one hand, it reduces manual operation and improves efficiency, and on the other hand, it can avoid possible errors in manual operation, which is conducive to improving accuracy.

[0052] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0054] Figure 1 This is a schematic flowchart illustrating an instrument identification method according to an embodiment of the present disclosure.

[0055] Figure 2 This is a schematic diagram of an encoder according to an embodiment of the present disclosure.

[0056] Figure 3 This is a schematic flowchart illustrating another instrument identification method according to an embodiment of the present disclosure.

[0057] Figure 4 This is a schematic diagram of a sub-encoder according to an embodiment of the present disclosure.

[0058] Figure 5 This is a schematic diagram of a feature pyramid according to an embodiment of the present disclosure.

[0059] Figure 6 This is a schematic block diagram illustrating an instrument identification device according to an embodiment of the present disclosure. Detailed Implementation

[0060] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0061] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0062] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of embodiments of this disclosure, a first target sub-model may also be referred to as a second target sub-model, and similarly, a second target sub-model may also be referred to as a first target sub-model. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0063] For the sake of brevity and ease of understanding, this document uses the terms "greater than" or "less than", "higher than" or "lower than" to describe size relationships. However, it will be understood by those skilled in the art that the term "greater than" also includes the meaning of "greater than or equal to", and "less than" also includes the meaning of "less than or equal to"; the term "higher than" also includes the meaning of "higher than or equal to", and "lower than" also includes the meaning of "lower than or equal to".

[0064] Figure 1 This is a schematic flowchart illustrating an instrument identification method according to an embodiment of this disclosure. The method shown in this embodiment can be applied to electronic devices equipped with a processor. The steps in the following embodiments can be mainly executed by the processor. The electronic device includes, but is not limited to, a terminal and a server. The terminal may be, for example, a mobile phone, a tablet computer, a wearable device, etc., and the server may be, for example, a local service, a cloud server, etc.

[0065] like Figure 1 As shown, the instrument identification method includes the following steps:

[0066] In step S101, the embedded features of pixels in the target instrument image are determined, and the position information of the pixels is encoded to obtain position encoded features. The target instrument image contains multiple key points related to the instrument pointer as labels.

[0067] In step S102, the superimposed feature obtained by superimposing the position encoding feature and the embedded feature is input into the encoder of the target model;

[0068] The input of the target model includes the label, and the output of the target model includes the coordinates of the key points in the sample instrument image;

[0069] The encoder includes a multi-head self-attention layer, and the target model is used to encode the superimposed features to obtain encoded features, and to determine the coordinates of the key points in the target instrument image based on the encoded features.

[0070] In one embodiment, multiple sample instrument images can be collected first, and then a training sample set can be constructed based on the sample instrument images. The target model can then be trained based on the training sample set.

[0071] For example, before inputting the superimposed feature obtained by superimposing the location-encoded feature and the embedded feature into the target model, the method further includes:

[0072] The embedded features of pixels in the sample instrument image are determined, and the sample position information of the pixels is encoded to obtain position encoding features. The sample instrument image contains multiple sample key points related to the instrument pointer as sample labels.

[0073] The sample superposition feature obtained by superimposing the sample location encoding feature and the sample embedded feature is input into the encoder to obtain the sample encoding feature;

[0074] The training sample set is determined based on sample coding features obtained from multiple sample instrument images;

[0075] The initial model is trained based on the sample encoding features in the training sample set to obtain the target model, wherein the sample model includes the encoder, the input of the sample model includes the label, and the output of the sample model includes at least the coordinates of the sample key points in the sample instrument image.

[0076] Based on this, the trained target model can determine the embedding features of pixels in a target instrument image that requires a reading, and can also encode the position information of pixels to obtain position encoded features.

[0077] For example, key points can be identified as labels in the target instrument image. These key points may include at least one of the following: the starting position of the dial reading, the ending position of the dial reading, the midpoint position of the dial reading, the starting position of the instrument pointer, and the ending position of the instrument pointer.

[0078] It should be noted that the key points selected as labels can be set as needed. The selection of the above 5 key points as labels in this embodiment is only one implementation method.

[0079] Next, the pixels in the target instrument image can be unfolded in one dimension (horizontally or vertically), and then the embedded features of the pixels can be calculated through a fully connected layer. For example, the dimension of the embedded features is d.

[0080] In order to determine the positional encoding features of a pixel, the expanded pixels can be numbered, and then the positional encoding features can be obtained by random initialization based on the pixel numbers. For example, the dimension of the positional encoding features is the same as the dimension of the embedded features, which is also d-dimensional.

[0081] Then, the positional encoding features and embedded features can be superimposed to obtain superimposed features, for example, by concatenating the positional encoding features and embedded features using the concat method.

[0082] Furthermore, the encoder of the target model can process the superimposed features to obtain one or more tensors. The target model can further process the tensors to output the coordinates of key points in the target instrument image. Then, the pointer reading in the target instrument image can be determined based on the obtained coordinates.

[0083] According to embodiments of this disclosure, the instrument image can be processed by the trained target model to output the coordinates of key points in the target instrument image. Then, the pointer reading in the target instrument image can be determined based on the obtained coordinates, thereby realizing the automatic determination of the pointer reading in the instrument based on the instrument image. On the one hand, it reduces manual operation and improves efficiency, and on the other hand, it can avoid possible errors in manual operation, which is conducive to improving accuracy.

[0084] Furthermore, the encoder of the target model includes a multi-head self-attention layer. This layer projects the three main parameters Q (query vector sequence), K (key vector sequence), and V (value vector sequence) from the attention layer through h different linear transformations, where h is the number of heads in the multi-head model. The multi-head self-attention layer maps Q and K to different subspaces of the higher-order space α to calculate similarity. The calculated results are normalized and multiplied with V as weights. Then, the attention information from different subspaces is concatenated (e.g., concat). This reduces the dimensionality of each vector when calculating the attention for each head, which helps avoid overfitting during the training of the target model.

[0085] Figure 2 This is a schematic diagram of an encoder according to an embodiment of the present disclosure.

[0086] like Figure 2As shown, the initial input to the encoder is Inputs, such as a target instrument image. Then, the embedded feature (Input Embedding) of pixels in the target instrument image can be determined, as well as the positional encoding feature (Positional Encoding) of the pixels. The superimposed feature obtained by superimposing the positional encoding feature and the embedded feature is then input into the encoder of the target model. The N to the left of the encoder indicates that N encoders can be connected together for multiple processing steps on the superimposed feature.

[0087] The encoder may include a multi-head attention layer, a feed forward layer, and an add & normalize layer. The output and input of the multi-head attention layer can be processed by the add & normalize layer and then input into the feed forward layer. The output and input of the feed forward layer can be processed by the add & normalize layer to obtain the encoded feature output.

[0088] Figure 3 This is a schematic flowchart illustrating another instrument identification method according to an embodiment of this disclosure. Figure 3 As shown, in one embodiment, determining the embedded features of pixels in the target instrument image and encoding the position information of the pixels to obtain position-coded features includes:

[0089] In step S301, the target instrument image is divided into multiple blocks;

[0090] In step S302, the embedded features of pixels in each image are determined, and the position information of the pixels is encoded to obtain position encoded features.

[0091] In one embodiment, with the improvement of photography and display technologies, the resolution of images is getting higher and higher, and the number of pixels in the images is also increasing. The instrument image in this disclosure can contain a large number of pixels. However, when a large number of pixels are processed together by an encoder, the resulting encoded features have relatively low accuracy in representing the semantics of the image.

[0092] Therefore, according to this embodiment, the target instrument image can first be divided into multiple blocks, such as multiple regular blocks or irregular blocks. Then, for each image block, one-dimensional unfolding can be performed in the same way to obtain embedded features. Taking regular blocks as an example, it can be divided into 9 blocks of 3×3, or 64 blocks of 8×8, etc.

[0093] This allows for the determination of embedded features and positional coding features for each image block, resulting in superimposed features. Since the semantics represented by the superimposed features are specific to a single image block and correspond to a relatively small number of pixels, the accuracy of the representation is relatively high. Consequently, the coding features obtained through encoder processing also have relatively high accuracy in representing the semantics of the image.

[0094] For cases where the target instrument image is divided into multiple blocks, the following example describes the general process of obtaining embedded features.

[0095] For example, the target instrument image can be divided into n blocks, namely X1 to X... n The corresponding positional encoding feature is E pos1 To E posn Each block passes through a fully connected layer E. d The embedded features are obtained, where d represents the dimension of the embedded features obtained after processing by the fully connected layer. The fully connected layer may include, for example, a linear liner layer, a linearly rectified ReLU layer (which can act as an activation function, or other activation functions can be used as needed), and a batch normalized (BNU) layer.

[0096] Therefore, for the target instrument image, the method to obtain the superimposed feature Z by concat is as follows:

[0097] Z = concat(X1E) d +E pos1 X2E d +E pos2 ,…,X n E d +E posn ).

[0098] For example, in the case of batch processing, the number of target instrument images processed each time is batchsize, which is greater than or equal to 1. Each target instrument image has the same size, which is 256*256, that is, the number of horizontal and vertical pixels is 256.

[0099] The size of the input target model image can be batchsize*3*256*256, where 3 represents the three color channels of a pixel, such as RGB (red, green, blue) channels. For example, if the target instrument image is divided into 8×8 blocks, resulting in 64 blocks, and the encoder's input dimension is 128, then the size can be varied.

[0100] First, transform batchsize*3*256*256 into batchsize*3*8*8*32*32, then further transform 32*32 into 128*1 through dimensionality reduction, and finally transform it into batchsize*3*64*128.

[0101] This process can be understood as a dimensionality reduction process. The encoded features obtained after the encoder can represent the semantics of each image and then be used to determine the label in each image. Since the label corresponds to the key point, the coordinates of the key point can be further determined after the label is determined.

[0102] In one embodiment, the encoder includes a plurality of sequentially connected sub-encoders, and the target model further includes a feature pyramid;

[0103] Each sub-encoder outputs an encoding result based on the input encoding features and inputs the encoding result into the feature pyramid to obtain multiple encoding features. The size information corresponding to the encoding features input to each sub-encoder is different, and the size information corresponding to the encoding result output by each sub-encoder is different.

[0104] Since a target instrument image has only one size, such as 256*256 in the above embodiment, the semantic information expressed by the encoded features determined for an image of only one size is relatively weak. To overcome this situation, the target instrument image can be processed, for example, by using multiple sub-encoders to process the encoded features of different size information separately, thereby obtaining multiple encoded results with different size information. Then, by comprehensively considering the multiple encoded results, the semantic information expressed is relatively strong.

[0105] However, while the resulting encoding has strong semantic information, it lacks positional information. Therefore, this embodiment further inputs the encoding results from each sub-encoder into the feature pyramid for processing, thereby enriching the positional information of the final output encoded features.

[0106] In one embodiment, each of the sub-encoders is connected to a linear layer, which is used to reduce the dimensionality of the encoding result output by the encoder and input the dimensionality-reduced encoding result into the next sub-encoder.

[0107] Figure 4 This is a schematic diagram of a sub-encoder according to an embodiment of the present disclosure.

[0108] In one embodiment, the description mainly focuses on the embedded feature portion of the encoding features. For example, the encoder includes three sub-encoders, namely encoder A, encoder B and encoder C. Encoder A is connected to the linear layer A, encoder B is connected to the linear layer B, and encoder C is connected to the linear layer C.

[0109] For example, the embedded feature of the input sub-encoder A is embedding(bsz,n,d), and the size information is (bsz,n,d). After processing by sub-encoder A, the size information of the encoded result stage1 is (bsz,n,d). Then, the dimensionality reduction of the encoded result stage1 is performed by the linear layer A, which can adjust the size information of the encoded result stage1, for example, adjusting it to (bsz,n / 4,d). The dimensionality-reduced encoded result can then be input into the next sub-encoder B.

[0110] Where bsz is an abbreviation for batchsize, n represents the target instrument image being divided into n blocks, and d represents the dimension of the embedded features.

[0111] The adjusted embedded feature embedding (bsz, n / 4, d) can then be input into sub-encoder B, with the size information being (bsz, n / 4, d). After processing by sub-encoder B, the size information of the encoded result stage2 is (bsz, n / 4, d). This stage2 is then subjected to dimensionality reduction processing by a linear layer B, which can adjust the size information of stage2, for example, to (bsz, n / 16, d). The dimensionality-reduced encoded result can then be input into the next sub-encoder C.

[0112] The adjusted embedded feature embedding (bsz, n / 16, d) can then be input into the sub-encoder C, with the size information (bsz, n / 16, d). After processing by the sub-encoder C, the size information of the encoded result stage3 is (bsz, n / 16, d). The dimensionality of the encoded result stage3 is then reduced by the linear layer C, which can adjust the size information of stage3, for example, to (bsz, n / 16, d). Encoder C is the last sub-encoder, and its output may not be connected to the linear layer.

[0113] Figure 5 This is a schematic diagram of a feature pyramid according to an embodiment of the present disclosure.

[0114] like Figure 5 As shown, the bottom-up processing of the target instrument image is equivalent to bottom-up feature convolution, which can be implemented through the multiple sub-encoders, for example, by... Figure 4 The three sub-encoders shown are implemented.

[0115] Furthermore, for the highest feature layer in the feature layer, such as Figure 5 The top layer on the left can be arranged according to... Figure 5 The process is then carried out from top to bottom. For example, upsampling can be performed to double the size of the features, and the number of channels can be changed by performing convolution on the lower layers (e.g., 1×1 convolution). Then, the features with the enlarged size and the features with the changed number of channels after convolution are added together.

[0116] And so on, such as Figure 5 As shown, three layers are defined from top to bottom. The output of each layer is called the prediction result. The output results of the three layers can be further concatenated through a fully connected layer to obtain the fused features. For example, the size information n, n / 4, and n / 16 are concatenated to obtain a dimension of (n+n / 4+n / 16) in the size information of the fused features.

[0117] Accordingly, positional features were spliced ​​into strong semantic information to ensure that the encoded features represent the instrument image more comprehensively and accurately.

[0118] In one embodiment, the output of the target model further includes at least one of the following:

[0119] The type of the target instrument image; the coordinates of at least two diagonal points of the instrument's circumscribed quadrilateral in the target instrument image.

[0120] In addition to outputting the coordinates of key points in the target instrument image, the target model can also be trained to output the type of the target instrument image and the coordinates of at least two diagonal points of the instrument's circumscribed quadrilateral in the target instrument image by constructing the initial model during training.

[0121] Specifically, based on the type of the target instrument image output by the target model, it can be determined whether the target instrument image is a meter image. If it does not contain any instruments, then the type is not a meter image. In this case, the processing of the target model can be paused to avoid wasting memory.

[0122] By determining the coordinates of at least two diagonal points of the instrument's circumscribed quadrilateral in the target instrument image output by the target type, the position of the instrument in the target image can be determined, which is beneficial for subsequent accurate determination of the coordinates of key points in the target instrument image.

[0123] In one embodiment, the target model includes a first target sub-model, a second target sub-model, and a third target sub-model;

[0124] The input of the first target sub-model includes the fusion features, and the output includes the type of the target instrument image;

[0125] The input of the second target sub-model includes the fused features, and the output includes the coordinates of at least two diagonal points of the circumscribed quadrilateral of the instrument in the target instrument image;

[0126] The input of the third target sub-model includes the fused features, and the output includes the coordinates of the key points in the target instrument image.

[0127] In one embodiment, multiple sub-models can be constructed to form the target model, such as a first target sub-model, a second target sub-model, and a third target sub-model.

[0128] The inputs to the three sub-models can be the aforementioned fused features, and the fully connected layers in the sub-models can be configured according to the desired output. For example:

[0129] The size information tensor1 of the fully connected layer in the first target sub-model is ((bsz,(n+n / 4+n / 16)*d,2), where the third dimension 2 indicates that two results can be output, such as 1 and 0. 1 indicates that the target instrument image contains an instrument, and 0 indicates that the target instrument image does not contain an instrument.

[0130] The size information tensor2 of the fully connected layer in the second target sub-model is ((bsz,(n+n / 4+n / 16)*d,4), where the third dimension 4 indicates that 4 results can be output. For example, each of the 4 results constitutes the two coordinates of the diagonal point mentioned above.

[0131] The size information tensor3 of the fully connected layer in the third target sub-model is ((bsz,(n+n / 4+n / 16)*d,10), where the third dimension 10 indicates that four results can be output. For example, each set of ten results constitutes the coordinates of the five key points mentioned above.

[0132] The first objective sub-model can implement a classification task, and the loss function type can be crossentropy; the second objective sub-model can implement a regression task, and the loss function type can be L1 loss; the third objective sub-model can implement a regression task, and the loss function type can be L1 loss.

[0133] It should be noted that the relationship between each encoded feature and the sub-model is not limited to the situation described in the above embodiments, and can be adjusted as needed.

[0134] Corresponding to the embodiments of the instrument identification method described above, the present disclosure also proposes an embodiment of an instrument identification device.

[0135] The embodiments of this disclosure also propose an instrument identification device, which can be applied to electronic devices equipped with a processor. The steps in the following embodiments can be mainly executed by the processor. The electronic device includes, but is not limited to, a terminal and a server. The terminal may be, for example, a mobile phone, a tablet computer, a wearable device, etc., and the server may be, for example, a local service, a cloud server, etc.

[0136] In one embodiment, the instrument identification device includes one or more processors configured to perform:

[0137] The embedded features of pixels in the target instrument image are determined, and the position information of the pixels is encoded to obtain position encoded features. The target instrument image contains multiple key points related to the instrument pointer as labels.

[0138] The superimposed feature obtained by superimposing the positional encoding feature and the embedded feature is input into the encoder of the target model;

[0139] The input of the target model includes the label, and the output of the target model includes the coordinates of the key points in the sample instrument image;

[0140] The encoder includes a multi-head self-attention layer, and the target model is used to encode the superimposed features to obtain encoded features, and to determine the coordinates of the key points in the target instrument image based on the encoded features.

[0141] In one embodiment, the processor is further configured to perform:

[0142] The embedded features of pixels in the sample instrument image are determined, and the sample position information of the pixels is encoded to obtain position encoding features. The sample instrument image contains multiple sample key points related to the instrument pointer as sample labels.

[0143] The sample superposition feature obtained by superimposing the sample location encoding feature and the sample embedded feature is input into the encoder to obtain the sample encoding feature;

[0144] The training sample set is determined based on sample coding features obtained from multiple sample instrument images;

[0145] The initial model is trained based on the sample encoding features in the training sample set to obtain the target model, wherein the sample model includes the encoder, the input of the sample model includes the label, and the output of the sample model includes at least the coordinates of the sample key points in the sample instrument image.

[0146] In one embodiment, the key points include at least one of the following: the starting position of the dial reading, the ending position of the dial reading, the midpoint position of the dial reading, the starting position of the instrument pointer, and the ending position of the instrument pointer.

[0147] In one embodiment, the processor is configured to execute:

[0148] Divide the target instrument image into multiple blocks;

[0149] The embedded features of pixels in each image are determined, and the position information of the pixels is encoded to obtain position-coded features.

[0150] In one embodiment, the encoder includes a plurality of sequentially connected sub-encoders, and the target model further includes a feature pyramid;

[0151] Each sub-encoder outputs an encoding result based on the input encoding features and inputs the encoding result into the feature pyramid to obtain fused features. The size information corresponding to the encoding result output by each sub-encoder is different, and the size information corresponding to the encoding features input to each sub-encoder is different.

[0152] In one embodiment, each of the sub-encoders is connected to a linear layer, which is used to reduce the dimensionality of the encoding result output by the encoder and input the dimensionality-reduced encoding result into the next sub-encoder.

[0153] In one embodiment, the output of the target model further includes at least one of the following: the type of the target instrument image; the coordinates of at least two diagonal points of the instrument circumscribed quadrilateral in the target instrument image.

[0154] In one embodiment, the target model includes a first target sub-model, a second target sub-model, and a third target sub-model;

[0155] The input of the first target sub-model includes the fusion features, and the output includes the type of the target instrument image;

[0156] The input of the second target sub-model includes the fused features, and the output includes the coordinates of at least two diagonal points of the circumscribed quadrilateral of the instrument in the target instrument image;

[0157] The input of the third target sub-model includes the fused features, and the output includes the coordinates of the key points in the target instrument image.

[0158] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant methods, and will not be elaborated upon here.

[0159] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0160] Embodiments of this disclosure also provide an electronic device, including: a processor; a memory for storing a computer program; wherein, when the computer program is executed by the processor, it implements the instrument identification method described in any of the above embodiments.

[0161] Embodiments of this disclosure also provide a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the steps of the instrument identification method described in any of the above embodiments.

[0162] Figure 6 This is a schematic block diagram illustrating a device 600 for meter identification according to embodiments of the present disclosure. For example, device 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0163] Reference Figure 6 The device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0164] Processing component 602 typically controls the overall operation of device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the reference signal receiving method described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0165] Memory 604 is configured to store various types of data to support the operation of device 600. Examples of such data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0166] Power supply component 606 provides power to the various components of device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 600.

[0167] Multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0168] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0169] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0170] Sensor assembly 614 includes one or more sensors for providing status assessments of various aspects of device 600. For example, sensor assembly 614 may detect the on / off state of device 600, the relative positioning of components such as the display and keypad of device 600, changes in the position of device 600 or a component of device 600, the presence or absence of user contact with device 600, the orientation or acceleration / deceleration of device 600, and temperature changes of device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0171] Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0172] In an exemplary embodiment, the apparatus 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the reference signal receiving method described above.

[0173] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of the device 600 to complete the aforementioned reference signal receiving method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0174] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0175] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0176] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] The methods and apparatus provided in the embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An instrument identification method characterized by comprising: The method comprises: determining an embedded feature of a pixel in a target instrument image, and encoding position information of the pixel to obtain a position encoding feature, wherein the target instrument image contains a plurality of key points related to instrument pointers as labels; the embedded feature is obtained by one-dimensional expansion of the pixels in the target instrument image and calculation through a fully connected layer; inputting a superimposed feature obtained by superimposing the position encoding feature and the embedded feature into an encoder of a target model; wherein the input of the target model comprises the labels, and the output of the target model comprises the coordinates of the key points in a sample instrument image; the encoder comprises a multi-head self-attention layer, and the target model is used to encode the superimposed feature to obtain an encoded feature, and determine the coordinates of the key points in the target instrument image according to the encoded feature.

2. The method of claim 1, wherein, Before the superimposed feature obtained by superimposing the position encoding feature and the embedded feature is input into the target model, the method further comprises: determining a sample embedded feature of a pixel in a sample instrument image, and encoding sample position information of the pixel to obtain a position encoding feature, wherein the sample instrument image contains a plurality of sample key points related to instrument pointers as sample labels; inputting a sample superimposed feature obtained by superimposing the sample position encoding feature and the sample embedded feature into the encoder to obtain a sample encoded feature; determining a training sample set based on the sample encoded features obtained from a plurality of sample instrument images; training an initial model according to the sample encoded features in the training sample set to obtain a target model, wherein the initial model comprises the encoder, the input of the initial model comprises the labels, and the output of the initial model at least comprises the coordinates of the sample key points in the sample instrument image.

3. The method of claim 1, wherein, The key points comprise at least one of: a dial reading start position, a dial reading end position, a dial reading midpoint position, an instrument pointer start position, and an instrument pointer end position.

4. The method of claim 1, wherein, The determination of the embedded feature of the pixel in the target instrument image and the encoding of the position information of the pixel to obtain the position encoding feature comprises: dividing the target instrument image into a plurality of blocks; determining the embedded feature of the pixel in each block image, and encoding the position information of the pixel to obtain the position encoding feature.

5. The method of claim 4, wherein, The encoder comprises a plurality of sub-encoders connected in sequence, and the target model further comprises a feature pyramid; each sub-encoder outputs an encoding result according to the input encoded feature, and inputs the encoding result into the feature pyramid to obtain a fused feature, wherein the size information corresponding to the encoding result output by each sub-encoder is different, and the size information corresponding to the encoded feature input into each sub-encoder is different.

6. The method of claim 5, wherein, Each sub-encoder is connected with a linear layer, which is used to reduce the dimension of the encoding result output by the encoder, and input the reduced encoding result into the next sub-encoder.

7. The method of claim 5, wherein, The output of the target model further comprises at least one of: a type of the target instrument image; and coordinates of at least two diagonal points of a four-vertex rectangle circumscribed around an instrument in the target instrument image.

8. The method of claim 7, wherein, The target model comprises a first target sub-model, a second target sub-model and a third target sub-model; The input of the first target sub-model comprises the fusion feature, and the output comprises the type of the target instrument image; The input of the second target sub-model comprises the fusion feature, and the output comprises the coordinates of at least two diagonal points in the instrument quadrilateral in the target instrument image; The input of the third target sub-model comprises the fusion feature, and the output comprises the coordinates of the key points in the target instrument image.

9. An apparatus for instrument identification, comprising: The processor is configured to perform: Determine the embedded feature of the pixel in the target instrument image, and encode the position information of the pixel to obtain the position encoding feature, wherein the target instrument image contains a plurality of key points related to instrument pointers as labels; the embedded feature is calculated by one-dimensional expansion of the pixel in the target instrument image and then passing through a fully connected layer; The superimposed feature obtained by superimposing the position encoding feature and the embedded feature is input into the encoder of the target model; The input of the target model comprises the label, and the output of the target model comprises the coordinates of the key points in the sample instrument image; The encoder comprises a multi-head self-attention layer, and the target model is used to encode the superimposed feature to obtain an encoded feature, and determine the coordinates of the key points in the target instrument image according to the encoded feature.

10. The apparatus of claim 9, wherein, The processor is further configured to perform: Determine the sample embedded feature of the pixel in the sample instrument image, and encode the sample position information of the pixel to obtain the position encoding feature, wherein the sample instrument image contains a plurality of sample key points related to instrument pointers as sample labels; The sample superimposed feature obtained by superimposing the sample position encoding feature and the sample embedded feature is input into the encoder to obtain a sample encoded feature; Determine a training sample set based on the sample encoded features obtained from a plurality of sample instrument images; Train an initial model based on the sample encoded features in the training sample set to obtain a target model, wherein the initial model comprises the encoder, the input of the initial model comprises the label, and the output of the initial model at least comprises the coordinates of the sample key points in the sample instrument image.

11. The apparatus of claim 9, wherein, The key points comprise at least one of: the starting point position of the dial reading, the ending point position of the dial reading, the midpoint position of the dial reading, the starting point position of the instrument pointer, and the ending point position of the instrument pointer.

12. The apparatus of claim 9, wherein, The processor is configured to perform: Divide the target instrument image into a plurality of blocks; Determine the embedded feature of the pixel in each block, and encode the position information of the pixel to obtain the position encoding feature.

13. The apparatus of claim 12, wherein, The encoder comprises a plurality of sub-encoders connected in sequence, and the target model further comprises a feature pyramid; Each sub-encoder outputs an encoding result according to the input encoded feature, and inputs the encoding result into the feature pyramid to obtain a fusion feature, wherein the size information corresponding to the encoding result output by each sub-encoder is different, and the size information corresponding to the encoded feature input into each sub-encoder is different.

14. The apparatus of claim 13, wherein, Each of the sub-encoders is connected with a linear layer, which is used to reduce the dimension of the encoding result output by the encoder and input the reduced encoding result into the next sub-encoder.

15. The apparatus of claim 13, wherein, The output of the target model further comprises at least one of: a type of the target instrument image; and 16. The apparatus of claim 15, wherein, coordinates of at least two diagonal points of a quadrilateral circumscribing the instrument in the target instrument image. The target model comprises a first target sub-model, a second target sub-model and a third target sub-model. The input of the first target sub-model comprises the fusion feature, and the output comprises the type of the target instrument image. The input of the second target sub-model comprises the fusion feature, and the output comprises the coordinates of at least two diagonal points of a quadrilateral circumscribing the instrument in the target instrument image.

17. An electronic device, comprising: The input of the third target sub-model comprises the fusion feature, and the output comprises the coordinates of the key points in the target instrument image. comprise: a processor; a memory for storing a computer program; 18. A computer readable storage medium for storing a computer program, characterized in that, wherein the computer program, when executed by the processor, implements the instrument identification method of any one of claims 1 to 8. the computer program, when executed by the processor, implements the steps of the instrument identification method of any one of claims 1 to 8.

Images