Instrument reading method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides an instrument reading method and device, electronic equipment and a computer readable storage medium. The method comprises the steps that corresponding instrument features are matched according to an instrument image; through the key point model, identifying target scale key point coordinates corresponding to the target scale in the instrument image and instrument center key point coordinates; the key point model is obtained by training target scale key points and instrument center key points of various types of instruments; performing multi-pointer key point detection on the instrument image to obtain a plurality of pointer key point coordinates of each pointer in the instrument image; and according to the plurality of pointer key point coordinates corresponding to each pointer, the target scale key point coordinates and the instrument characteristics, the reading of the corresponding instrument is determined. According to the method, key point model training is carried out on the target scale key points and the instrument center key points of various types of instruments, so that the identified instrument image types are increased, and the universality of the instrument reading method is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an instrument reading method, device, electronic device and computer-readable storage medium. Background Art

[0002] The general process of commonly used instrument readings at present is to obtain the instrument panel area through the target detection model or semantic segmentation model, use the segmentation model or instance segmentation model to obtain the pointer area of ​​the instrument panel area, and then read the pointer reading. Although this method can realize the reading of pointer readings to a certain extent, it is usually only applicable to a single type of instrument and has poor versatility. Summary of the invention

[0003] In view of this, an object of the embodiments of the present application is to provide an instrument reading method, an apparatus, an electronic device, and a computer-readable storage medium, which can improve the versatility of the instrument reading method.

[0004] In a first aspect, an embodiment of the present application provides an instrument reading method, comprising: matching corresponding instrument features according to an instrument image; identifying target scale key point coordinates and instrument center key point coordinates corresponding to the target scale in the instrument image through a key point model; wherein the key point model is obtained by training the target scale key points and the instrument center key points of multiple types of instruments; performing multi-pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image; and determining the reading of the corresponding instrument based on the multiple pointer key point coordinates corresponding to each pointer, the target scale key point coordinates and the instrument features.

[0005] In the above implementation process, by training the key point model based on the target scale key points and instrument center key points of various types of instruments, the obtained key point model can perform target scale key point coordinate recognition and instrument center key point coordinate recognition for instrument images of various types of instruments, which can increase the types of instrument images recognized by the key point model and improve the versatility of the instrument reading method. In addition, the instrument reading is determined by the instrument range, target scale key point coordinates, instrument center key point coordinates, and pointer key point coordinates in the instrument image, and can be calculated directly based on the relationship between the coordinates without being restricted by the instrument type, further improving the versatility of the instrument reading method.

[0006] In one embodiment, matching the corresponding instrument features according to the instrument image includes: obtaining the normalized features of the instrument image through a preset model; and determining the instrument identity in the instrument library that matches the instrument image according to the normalized features and the instrument features of each instrument in the instrument library; wherein the instrument library is configured to store instrument features of multiple instruments, the instrument features including instrument ranges; and when the instrument image matches the corresponding instrument identity, the instrument features corresponding to the instrument identity are used as the instrument features of the instrument type.

[0007] In the above implementation process, an instrument library is set up, and different instrument features of different instrument types are stored in the instrument library. When obtaining the instrument features corresponding to the instrument image, the normalized features of the instrument in the instrument image are matched with various instrument features in the instrument library, and then the corresponding instrument features such as the range and scale angle are obtained. The determination of the instrument features is not affected by lighting, camera angle, etc., which improves the accuracy of the instrument features.

[0008] In one embodiment, the preset model uses the ViT-Tiny as a backbone network; Center Loss is used as a loss function for training the preset model.

[0009] In the above implementation process, since Center Loss is optimized by combining the distance of the feature center, by using Center Loss as the loss function of the preset model to train the preset model, the discrimination ability of the model learning features and the generalization performance of the model can be effectively improved. Since the ViT-Tiny operation can be analogous to face recognition, when the Center Loss is used to train ViT-Tiny, it can effectively deal with the problem of out-of-set samples, simplify the training process, and improve training efficiency. In addition, by using ViT-Tiny as the backbone network of the preset model for feature matching, the matching accuracy can be improved, and the model has strong generalization ability, and the preset model is more practical and robust, and has less overfitting.

[0010] In one embodiment, performing multi-pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image includes: performing multi-pointer key point detection on the instrument image through a RepViT-yolov8-pose model to obtain multiple pointer key point coordinates of each pointer in the instrument image; wherein the multiple pointer key point coordinates of each pointer include a pointer root coordinate and a pointer tip coordinate of each pointer; wherein the RepViT-yolov8-pose model includes key point loss weights and label smoothing.

[0011] In the above implementation process, since the RepViT-yolov8-pose model combines the characteristics of RepViT's fast speed and high generalization ability with the characteristics of yolov8-pose model's high recognition accuracy and stability, the RepViT-yolov8-pose model is used to identify the key points of the pointer, and the key point information of one or more pointers in the instrument can be accurately identified. It is no longer identified based on color, length, etc., is less affected by lighting, and has high recognition accuracy and stability.

[0012] In one embodiment, before matching the corresponding instrument features according to the instrument image, the method also includes: acquiring the instrument image through the yolov8-repvit target detection model; wherein the backbone network of the yolov8-repvit target detection model is the RepViT lightweight Transformer.

[0013] In the above implementation process, by setting the backbone network of the yolov8-repvit target detection model to the RepViT lightweight Transformer, the accuracy and efficiency of instrument image acquisition can be improved while improving the generalization ability of the model.

[0014] In one embodiment, before matching the corresponding instrument features according to the instrument image, the method also includes: predicting the rotation angle of the instrument image through an angle regression network based on RepViT; rotating the instrument image along the image center according to the rotation angle; preprocessing the rotated instrument image; identifying the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the instrument image through a key point model, including: identifying the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the preprocessed rotated instrument image through a key point model; performing multi-pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image, including: performing multi-pointer key point detection on the preprocessed rotated instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image.

[0015] In the above implementation process, after the instrument image is acquired, the offset angle of the instrument image is predicted based on the angle regression network, and then the instrument image is affine transformed and rotated according to the offset angle to rotate the instrument image and obtain a more accurate instrument image, thereby improving the accuracy of the instrument image.

[0016] In one embodiment, the instrument features include a target scale angle and an instrument range, wherein the target scale angle is an angle of the target scale relative to a starting time;

[0017] Before determining the reading of the corresponding instrument according to the coordinates of the multiple pointer key points corresponding to each pointer, the target scale key points and the instrument features, the method also includes: establishing a polar coordinate system based on the distance between the target scale and the center of the instrument, and the target scale angle; mapping the polar coordinate system to a Cartesian coordinate system to obtain the converted target scale key point coordinates; determining a perspective transformation matrix according to the converted target scale key point coordinates and the target scale key point coordinates before the conversion; performing perspective transformation on the rotated instrument image based on the perspective transformation matrix; determining the reading of the corresponding instrument according to the coordinates of the multiple pointer key points corresponding to each pointer, the target scale key points and the instrument features includes: determining the reading of the corresponding instrument according to the multiple pointer key point coordinates after the perspective transformation, the target scale key point coordinates after the perspective transformation and the instrument range.

[0018] In the above implementation process, since perspective transformation can change the problem of instrument imaging deformation caused by camera angle affecting reading, a polar coordinate system is established based on the angle of the target scale key point and the distance between the target scale and the center of the instrument, and the perspective transformation matrix is ​​solved by mapping the polar coordinate system to the Cartesian coordinate system, thereby realizing perspective correction of the instrument image, which is applicable to various types of instruments and provides more accurate readings. It is not only compatible with the application scenarios of multiple types of pointers such as single pointer and multiple pointers, but also can improve the accuracy and precision of pointer recognition.

[0019] In a second aspect, an embodiment of the present application also provides an instrument reading device, comprising: a type determination module, used to match the corresponding instrument features according to the instrument image; an identification module, used to identify the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the instrument image through a key point model; wherein the key point model is obtained by training the target scale key points and the instrument center key points of multiple types of instruments; a detection module, used to perform multi-pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image; a reading determination module, used to determine the reading of the corresponding instrument based on the multiple pointer key point coordinates corresponding to each pointer, the target scale key point coordinates and the instrument features.

[0020] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method in the above-mentioned first aspect, or any possible implementation of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the instrument reading method in the above-mentioned first aspect or any possible implementation manner of the first aspect are executed.

[0022] In order to make the above-mentioned objects, features and advantages of the present application more obvious and understandable, embodiments are given below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of the interaction between the electronic device and the image acquisition device provided in an embodiment of the present application;

[0025] Figure 2 A block diagram of an electronic device provided in an embodiment of the present application;

[0026] Figure 3 A flow chart of an instrument reading method provided in an embodiment of the present application;

[0027] Figure 4 A comparison diagram of the instrument image before and after rotation provided in the embodiment of the present application;

[0028] Figure 5 A specific flow chart of meter reading provided for the embodiment of the present application;

[0029] Figure 6 A schematic diagram of the functional modules of the meter reading device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0031] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0032] In the field of industrial automation and detection technology, instruments are key equipment for data acquisition and status monitoring. The versatility of their reading methods is of great significance for improving production efficiency, reducing maintenance costs and promoting system integration.

[0033] However, the inventors of the present application have found through long-term research that the meter reading methods widely used in the current market generally still have the following problems:

[0034] (1) In actual application scenarios, there are many types of meters, such as round meters, C-type meters, and semi-C-type meters. Current meter reading methods usually only support a single type of meter, which has poor versatility.

[0035] (2) The current instrument reading method does not have an efficient solution for uniformly identifying single pointers and multiple pointers;

[0036] (3) There is no solution to the low reading accuracy caused by angle problems of C-type and semi-C-type meters.

[0037] In view of this, the present application proposes an instrument reading method, which trains a key point model based on the target scale key points and instrument center key points of various types of instruments, so that the obtained key point model can perform target scale key point coordinate recognition and instrument center key point coordinate recognition for instrument images of various types of instruments, which can increase the types of instrument images recognized by the key point model and improve the versatility of the instrument reading method. In addition, the instrument reading is determined by the instrument range, target scale key point coordinates, instrument center key point coordinates, and pointer key point coordinates in the instrument image, and can be calculated directly based on the relationship between the coordinates without being restricted by the instrument type, further improving the versatility of the instrument reading method.

[0038] To facilitate understanding of this embodiment, the operating environment of an instrument reading method disclosed in the embodiment of the present application is first introduced in detail.

[0039] like Figure 1 , which is a schematic diagram of the interaction between the image acquisition device 200 and the electronic device 100 provided in the embodiment of the present application. The electronic device 100 is connected to one or more image acquisition devices 200 through a network for data communication or interaction. The electronic device 100 can be a network server, a database server, a personal computer (PC), a tablet computer, a smart phone, a personal digital assistant (PDA), etc., and the electronic device 100 can be selected according to actual conditions.

[0040] The image acquisition device 200 is configured to acquire an instrument image and transmit the acquired instrument image to the electronic device 100. After acquiring the instrument image, the electronic device 100 processes the instrument image and reads the reading of the instrument in the instrument image based on the processed instrument image.

[0041] Optionally, the image acquisition device 200 may be a camera, a webcam, a mobile phone, a tablet computer, etc. The image acquisition device 200 may be selected according to actual conditions.

[0042] To facilitate understanding of this embodiment, the electronic device 100 that executes the meter reading method disclosed in the embodiment of the present application is described in detail below.

[0043] like Figure 2 , which is a block diagram of an electronic device. The electronic device 100 may include a memory 111 and a processor 113. A person skilled in the art may understand that Figure 2 The structure shown is only for illustration and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations shown.

[0044] The memory 111 and the processor 113 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute the executable module stored in the memory.

[0045] The memory 111 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 113 executes the program after receiving the execution instruction. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application can be applied to the processor 113, or implemented by the processor 113.

[0046] The processor 113 may be an integrated circuit chip with signal processing capability. The processor 113 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0047] The electronic device 100 in this embodiment can be used to execute each step in each method provided in the embodiments of the present application. The implementation process of the meter reading method is described in detail below through several embodiments.

[0048] See also Figure 3 , is a flow chart of the meter reading method provided in the embodiment of the present application. Figure 3 The specific process shown is described in detail.

[0049] Step 201, matching corresponding instrument features according to the instrument image.

[0050] The instrument image refers to an image containing the instrument acquired by the image acquisition device.

[0051] The meter type here refers to the type of meter. For example, round meter, C-type meter, semi-C-type meter, etc. The meter type can be selected according to actual conditions.

[0052] The above-mentioned instrument characteristics refer to characteristic parameters in the instrument, such as instrument range, target scale key points, instrument center, and normalized characteristics.

[0053] In one embodiment, one or more parameters such as instrument range, target scale key point, instrument center, etc. corresponding to various instrument types may be different.

[0054] Optionally, before step 201, the instrument features of each instrument type may be bound to the instrument type. After the instrument type in the instrument image is determined, the corresponding instrument features may be matched directly according to the instrument type. That is, step 201 includes: obtaining the normalized features of the instrument image; and determining the instrument identity in the instrument library that matches the instrument type according to the normalized features and the instrument features of each instrument in the instrument library; when the instrument type matches the corresponding instrument identity, the instrument features corresponding to the instrument identity are used as the instrument features of the instrument type.

[0055] Step 201 may also include: directly acquiring normalized features of the instrument in the instrument image, matching the normalized features with instrument features in the instrument library, and then determining an instrument identity that matches the instrument image, and determining that the instrument features corresponding to the matched instrument identity are the instrument features of the instrument image.

[0056] The above-mentioned method for determining the instrument characteristics is only exemplary, and the method for obtaining the instrument characteristics can be determined according to actual conditions.

[0057] Step 202: using a key point model, identifying the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the instrument image.

[0058] Among them, the key point model is obtained by training the target scale key points and instrument center key points of various types of instruments.

[0059] The target scale here can be a long scale in the instrument, a short scale in the instrument, or all scales in the instrument. The target scale can be selected according to actual conditions.

[0060] The instrument center mentioned above refers to the rotation center of the instrument pointer.

[0061] Exemplarily, if the instrument types include circular instruments, C-type instruments, semi-C-type instruments and the like, after building the key point model framework, the key point model framework is trained together with target scale key points and instrument center key points of multiple circular instruments, target scale key points and instrument center key points of multiple C-type instruments, and target scale key points and instrument center key points of multiple semi-C-type instruments to obtain a trained key point model.

[0062] In one embodiment, if there are differences in the number of target scale key points of multiple instrument types, key point model training may be performed according to the number of target scale key points corresponding to the instrument type with the largest number of target scale key points.

[0063] For example, if the number of target scale key points of a circular instrument is 17, the number of target scale key points of a C-type instrument is 10, and the number of target scale key points of a semi-C-type instrument is 6, then 17 target scale key points of each instrument type can be extracted for key point model training. Among them, the number of target scale key points of a C-type instrument is 10, which does not meet the 17 target scale key points. Starting from the 11th target scale key point, the remaining target scale key points are supplemented by 0. The number of target scale key points of a semi-C-type instrument is 6, which does not meet the 17 target scale key points. Starting from the 7th target scale key point, the remaining target scale key points are supplemented by 0. After the supplementation, the number of target scale key points of the C-type instrument and the semi-C-type instrument reaches 17, and the key point model training is performed with the supplemented 17 target scale key points.

[0064] It can be understood that since the key point model is obtained by training multiple instrument types, after the instrument image is input into the key point model, the key point model can output the target scale key point coordinates corresponding to the target scale of the instrument in the instrument image and the instrument center key point coordinates.

[0065] Step 203: perform multi-pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image.

[0066] The multiple pointer key points here can be multiple key points of a pointer. For example, the main pointer root key point, the main pointer tip key point, etc. The multiple pointer key points can also be multiple key points of multiple pointers. For example, the main pointer root key point, the main pointer tip key point, the auxiliary pointer root key point, the auxiliary pointer tip key point, etc. The multiple pointer key points can be selected according to actual conditions.

[0067] Step 204, determining the reading of the corresponding instrument according to the coordinates of the multiple pointer key points corresponding to each pointer, the coordinates of the target scale key points and the instrument characteristics.

[0068] In one embodiment, the instrument feature includes the instrument range. Then step 204 can be implemented in the following manner:

[0069] Determine the instrument reading represented by each target scale key point according to the coordinates of the target scale key point and the instrument range, and determine the two target scale key points adjacent to the pointer according to the multiple pointer key points of the pointer. Calculate the first angle between the line from the pointer tip key point to the center of the instrument and the line from an adjacent target scale key point to the center of the instrument, calculate the second angle between the line from an adjacent target scale key point to the center of the instrument and the line from another adjacent target scale key point to the center of the instrument, and determine the reading pointed to by the pointer according to the ratio of the first angle to the second angle.

[0070] When there are multiple pointers in the instrument, the readings pointed by each pointer can be read according to the above process for each pointer. Finally, the reading of the instrument is determined according to the readings pointed by the multiple pointers.

[0071] In the above implementation process, by training the key point model based on the target scale key points and instrument center key points of various types of instruments, the obtained key point model can perform target scale key point coordinate recognition and instrument center key point coordinate recognition for instrument images of various types of instruments, which can increase the types of instrument images recognized by the key point model and improve the versatility of the instrument reading method. In addition, the instrument reading is determined by the instrument range, target scale key point coordinates, instrument center key point coordinates, and pointer key point coordinates in the instrument image, and can be calculated directly based on the relationship between the coordinates without being restricted by the instrument type, further improving the versatility of the instrument reading method.

[0072] In a possible implementation, step 201 includes: obtaining normalized features of the instrument image through a preset model; and determining an instrument identity in the instrument library that matches the instrument image based on the normalized features and instrument features of each instrument in the instrument library; when the instrument image matches the corresponding instrument identity, the instrument features corresponding to the instrument identity are used as instrument features of the instrument type.

[0073] The instrument library is configured to store instrument features of multiple instruments, including instrument range, scale angle, etc. The scale angle refers to the angle between the key point of the scale and the line connecting the center of the instrument and the starting scale.

[0074] The instrument library here can store instrument features corresponding to various instrument types.

[0075] The instrument characteristics may include a target scale angle and an instrument range, wherein the target scale angle is the angle of the target scale relative to the starting time.

[0076] In one embodiment, determining the instrument identity in the instrument library that matches the instrument type according to the normalized feature and the instrument features of each instrument in the instrument library can be implemented in the following manner:

[0077] The normalized feature and the instrument features of each instrument in the instrument library are similarly calculated to obtain the instrument feature with the highest similarity to the normalized feature, and the instrument identity corresponding to the instrument feature with the highest similarity is determined as the instrument identity matched to the instrument type.

[0078] It should be understood that in order to improve the matching accuracy, when performing instrument identification on the instrument image, a similarity threshold may be set, that is, the instrument identification matching is performed only when the similarity is within the similarity threshold range.

[0079] In one embodiment, if the highest similarity is within the similarity threshold range, the instrument identity corresponding to the instrument feature with the highest similarity is determined as the instrument identity matched by the instrument image.

[0080] If the highest similarity is not within the threshold range, similarity matching is not performed on the instrument image, and the instrument image is not matched to the corresponding instrument identity.

[0081] In the case where the instrument image is not matched to a corresponding instrument identity, the instrument image is registered with the corresponding instrument feature in the instrument library, and the instrument library stores the registered instrument type and instrument feature.

[0082] In the above implementation process, an instrument library is set up, and different instrument features of different instrument types are stored in the instrument library. When obtaining the instrument features corresponding to the instrument image, the normalized features of the instrument image are matched with the various instrument features in the instrument library, and then the corresponding instrument features such as the measuring range and scale angle are obtained. The determination of the instrument features is not affected by lighting, camera angle, etc., which improves the accuracy of the instrument features.

[0083] In a possible implementation, the preset model uses ViT-Tiny as the backbone network;

[0084] Among them, Center Loss is used as the loss function for training the preset model.

[0085] The ViT-Tiny here is a lightweight visual Transformer model. ViT-Tiny utilizes the structure of the Visual Transformer (ViT). This ViT-Tiny has the advantages of being lightweight and having relatively few parameters.

[0086] The Center Loss function mentioned above is a supervised learning loss function used in deep learning and computer vision tasks, especially face recognition and image classification. Its main purpose is to improve the classification performance and generalization ability of the model by reducing the distance between sample feature vectors in the same category while maintaining the separability of sample feature vectors between different categories during the training process.

[0087] In one embodiment, the formula of the Center Loss loss function may be as follows:

[0088]

[0089] Among them, L s is the loss function of the first module, L cis the loss function of the second module, L is the loss function, λ is 0.1, is the corresponding category y i The weight of x is the feature vector of the sample, is the corresponding category y i The bias of y i is the category label of the i-th sample.

[0090] In the above implementation process, since Center Loss is optimized by combining the distance of the feature center, by using Center Loss as the loss function of the preset model to train the preset model, the discrimination ability of the model learning features and the generalization performance of the model can be effectively improved. Since the ViT-Tiny operation can be compared to face recognition, when the Center Loss is used to train ViT-Tiny, it can effectively deal with the problem of out-of-set samples, simplify the training process, and improve training efficiency. In addition, by using ViT-Tiny as the backbone network of the preset model to perform feature matching, the matching accuracy can be improved, and the model has strong generalization ability, and the preset model is more practical and robust, and has less overfitting.

[0091] In a possible implementation, step 203 includes: performing multi-pointer key point detection on the instrument image through the RepViT-yolov8-pose model to obtain multiple pointer key point coordinates of each pointer in the instrument image.

[0092] The multiple pointer key point coordinates of each pointer include the pointer root coordinates and the pointer tip coordinates of each pointer.

[0093] Here, the RepViT-yolov8-pose model is an improved yolov8-pose model, which is a model that combines the RepViT architecture and the yolov8-pose architecture. The RepViT-yolov8-pose model includes key point loss weights and label smoothing.

[0094] Among them, the yolov8-pose model is a deep learning model that combines the YOLOv8 target detection algorithm with human pose estimation technology. It is used to achieve high-precision human pose recognition and key point detection in images or videos.

[0095] The yolov8-pose model integrates yolov8's powerful target detection capabilities and human pose estimation technology to accurately identify the human body in images or videos and precisely locate its key points, such as joints and limbs.

[0096] It should be understood that by using the yolov8-pose model for multi-pointer key point detection, analogizing the pointer key points with the human body posture key points, and training with multiple instrument images, a yolov8-pose model capable of identifying pointer key points is obtained.

[0097] The above-mentioned RepViT model is a model based on the Transformer architecture. The RepViT model can use the self-attention mechanism to process input data (such as images or video frames) and generate a series of representative vectors.

[0098] By combining RepViT with yolov8-pose to form a RepViT-yolov8-pose model that integrates the characteristics of the RepViT model and the yolov8-pose model, the generalization ability and recognition accuracy of the model can be improved without affecting the speed.

[0099] In one embodiment, the loss function of the RepViT-yolov8-pose model can be expressed as follows:

[0100] L=λ1L box +λ2L cls +λ3L pose +λ4L kobj ;

[0101] Among them, L is the total loss, λ1, λ2, λ3, λ4 are weights, and L box is the bounding box regression loss, L cls is the classification loss, L pose is the key point loss, L kobj is the key point target loss.

[0102] In the above implementation process, since the RepViT-yolov8-pose model combines the characteristics of RepViT's fast speed and high generalization ability with the characteristics of yolov8-pose model's high recognition accuracy and stability, the RepViT-yolov8-pose model is used to identify the key points of the pointer, and the key point information of one or more pointers in the instrument can be accurately identified. It is no longer identified based on color, length, etc., is less affected by lighting, and has high recognition accuracy and stability.

[0103] In a possible implementation, before step 201, the method further includes: acquiring an instrument image through a yolov8-repvit target detection model.

[0104] Among them, the backbone network of the yolov8-repvit target detection model is the RepViT lightweight Transformer.

[0105] The yolov8-repvit target detection model here is a computer vision model used for target detection tasks. The yolov8-repvit target detection model uses a lightweight network and can run smoothly on the device.

[0106] In the above implementation process, by setting the backbone network of the yolov8-repvit target detection model to the RepViT lightweight Transformer, the accuracy and efficiency of instrument image acquisition can be improved while improving the generalization ability of the model.

[0107] In a possible implementation, before step 201, the method further includes: predicting the rotation angle of the instrument image through an angle regression network based on RepViT; rotating the instrument image along the image center according to the rotation angle; and preprocessing the rotated instrument image.

[0108] In one embodiment, step 202 includes: identifying the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the pre-processed rotated instrument image through the key point model.

[0109] Step 203 includes: performing multi-pointer key point detection on the pre-processed and rotated instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image.

[0110] The RepViT-based angle regression network here can be a network that adds some task-specific layers (such as fully connected layers or convolutional layers) to RepViT to output the predicted value of angle regression.

[0111] In one embodiment, a loss function may also be set in the angle regression network to evaluate the difference between the predicted value and the true value and optimize it during training.

[0112] It should be understood that due to the placement and angle of the instrument, the acquisition angle of the image acquisition device, etc., the acquired instrument image may differ from the target angle, target direction, etc. When the angle, direction, etc. of the acquired instrument image differ from the target angle, target direction, etc., the subsequent identification of the pointer key points, scale key points, etc. of the instrument image may affect the recognition accuracy. By setting an angle regression network, after acquiring the instrument image, the instrument image is input into the angle regression network, and the angle regression network outputs the rotation angle that the instrument image needs to be rotated. Then, the instrument image is rotated according to the output rotation angle, so that the angle and direction of the rotated instrument image can be the target angle and target direction, and a more accurate and detailed instrument image can be obtained.

[0113] For example, if the instrument image is input into the angle regression network, the rotation angle output by the angle regression network is θ, where θ∈[0,2π); and the width and height of the obtained instrument image are W and H respectively, the width and height of the rotated instrument image are W0 and H0, and the instrument image coordinates are (X o , Y0), the coordinates of the rotated instrument image are (x, y), then the mathematical formula is:

[0114]

[0115] H0=Hcosθ+Wsinθ;

[0116] W0=W cosθ+H sinθ;

[0117] The above-mentioned preprocessing may include detection processing, cropping processing, etc., and the preprocessing method can be selected according to actual conditions.

[0118] It is understandable that after the instrument image is rotated, some redundant parts may appear in the image, which not only increases the memory of the instrument image, but also may affect the recognition of instrument pointer key points, scale key points, etc.

[0119] Therefore, after rotating the instrument image, the instrument image can be detected to determine whether there is a redundant part in the instrument image. If it is detected that there is a redundant part in the rotated instrument image, the redundant part can be cropped.

[0120] For example, Figure 4 As shown, the instrument image before rotation is a rectangle, and the instrument image after rotation is a diamond. In the diamond-shaped instrument image, all areas except the area where the instrument is located are redundant areas. These redundant areas can be cropped out so that the image for subsequent scale key point recognition and pointer key point recognition is an instrument image that only contains the instrument, thereby reducing recognition intervention.

[0121] In the above implementation process, after the instrument image is acquired, the offset angle of the instrument image is predicted based on the angle regression network, and then the instrument image is affine transformed and rotated according to the offset angle to rotate the instrument image and obtain a more accurate instrument image, thereby improving the accuracy of the instrument image.

[0122] In a possible implementation, before step 204, the method further includes: establishing a polar coordinate system based on the distance between the target scale and the center of the instrument, and the target scale angle; mapping the polar coordinate system to the Cartesian coordinate system to obtain the converted target scale key point coordinates; determining the perspective transformation matrix based on the converted target scale key point coordinates and the target scale key point coordinates before the conversion; and performing perspective transformation on the rotated instrument image based on the perspective transformation matrix.

[0123] Exemplarily, the perspective change can be achieved by:

[0124] Assume that the instrument image has N target scales, and the target scale angles of the N target scales are [θ1θ2...θ N ], the key point coordinates corresponding to the N target scale values ​​are [P1P2...P N ], the coordinate of the instrument center point is P0.

[0125] Since the target scale angle is the angle of each target scale relative to the starting scale, it is necessary to calculate the angle θ0 between the direction vector from the center of the instrument to the key at the starting moment and the X-axis direction of the pixel coordinate. The solution formula for θ0 is as follows:

[0126]

[0127] Furthermore, each target scale angle needs to be added with θ0 to obtain [θ′1θ′2...θ′ N ], calculate the distance between each target scale and the center of the instrument: that is, [r1r2...r N ]. The calculation formula for the distance between the target scale and the center of the instrument is as follows:

[0128]

[0129] Among them, the maximum length is the polar diameter of the polar coordinate system, that is, the formula for determining the polar diameter is as follows:

[0130] r=max{r1r2...r N};

[0131] Then the polar coordinate system coordinates of each target scale are:

[0132] [(r,θ′1)(r,θ′2)...(r,θ′ N )];

[0133] Convert the polar coordinate system to the Cartesian coordinate system, and the corrected coordinates are:

[0134] [P1′P2′...P′ N ];

[0135] Assuming that the coordinates of P1′ are (x′1y′1) and the coordinates of P0 are (x0y0), the conversion formula can be as follows:

[0136] x′1=x0+r cosθ;

[0137] y′1=y0+r sinθ;

[0138] According to the coordinates before transformation [P1P2...P N ] and the transformed coordinates, [P1′P2′...P′ N ], we can get the perspective transformation matrix H, the specific formula is as follows:

[0139]

[0140] It should be understood that after the perspective transformation matrix is ​​determined, the corrected image can be obtained based on the perspective transformation matrix. While the instrument image is perspective transformed, the pointer key points and the scale key points are also perspective transformed.

[0141] In one embodiment, step 204 includes: determining the reading of the corresponding instrument according to the coordinates of the multiple pointer key points after perspective transformation, the coordinates of the target scale key points after perspective transformation, and the instrument range.

[0142] Among them, step 202 and step 203 perform key point recognition based on the instrument image after perspective transformation, obtain the scale key point coordinates after perspective transformation and the pointer key point coordinates after perspective transformation. The final instrument reading is determined according to the scale key point coordinates after perspective transformation, the pointer key point coordinates after perspective transformation and the instrument range.

[0143] Specifically, the meter reading can be calculated in the following way:

[0144] Fit the pointer straight line according to the key points of the pointer root and the pointer tip, determine the intersection point C0 between the scale straight lines of the two target scales closest to the pointer straight line and the pointer straight line, and determine the scale key point on the left of the pointer straight line as C L , the key point on the scale to the right of the pointer line is C R , the coordinates of the instrument center point are P0. Then calculate and The angle θ L ,calculate and The angle θ T , C L and C R The corresponding range value is R L and R R , then the calculation formula for the meter reading can be as follows:

[0145]

[0146] In order to facilitate the understanding of the embodiments of this scheme, Figure 5As an example, the specific process of the instrument reading method of the embodiment of the present application is further described: First, the instrument image is obtained through the yolov8-repvit target detection model. Then the instrument image is input into the angle regression network, the rotation angle of the instrument image is predicted, and the instrument image is rotated according to the rotation angle, and then the instrument image is preprocessed. Match the corresponding instrument features based on the preprocessed instrument image, and if the corresponding instrument features are not matched, register the instrument features corresponding to the instrument image into the instrument library. In the case of matching the corresponding instrument features, obtain the matched instrument features, and identify the pointer key points, target scale key points and instrument center key points in the instrument image. Then, according to the identification of the pointer key points, target scale key points and instrument center key points, the target scale angle and the instrument range, the instrument image is perspective transformed. Finally, the instrument reading is read based on the perspective transformed instrument features, pointer key points, target scale key points and instrument center key points.

[0147] In the above implementation process, since perspective transformation can change the problem of instrument imaging deformation caused by camera angle affecting reading, a polar coordinate system is established based on the angle of the target scale key point and the distance between the target scale and the center of the instrument, and the perspective transformation matrix is ​​solved by mapping the polar coordinate system to the Cartesian coordinate system, thereby realizing perspective correction of the instrument image, which is applicable to various types of instruments and provides more accurate readings. It is not only compatible with the application scenarios of multiple types of pointers such as single pointer and multiple pointers, but also can improve the accuracy and precision of pointer recognition.

[0148] Based on the same application concept, an instrument reading device corresponding to the instrument reading method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to that in the aforementioned instrument reading method embodiment, the implementation of the device in this embodiment can refer to the description in the embodiment of the aforementioned method, and the repeated parts will not be repeated.

[0149] See also Figure 6 , is a functional module diagram of the meter reading device provided in the embodiment of the present application. Each module in the meter reading device in this embodiment is used to execute each step in the above method embodiment. The meter reading device includes a type determination module 301, an identification module 302, a detection module 303, and a reading determination module 304; wherein,

[0150] The type determination module 301 is used to match corresponding instrument features according to the instrument image.

[0151] The recognition module 302 is used to identify the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the instrument image through a key point model; wherein the key point model is obtained by training the target scale key points and the instrument center key points of multiple types of instruments.

[0152] The detection module 303 is used to perform multiple pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image.

[0153] The reading determination module 304 is used to determine the reading of the corresponding instrument according to the coordinates of the multiple pointer key points corresponding to each pointer, the coordinates of the target scale key points and the instrument characteristics.

[0154] In a possible implementation, the type determination module 301 is further used to: obtain normalized features of the instrument image through a preset model; and determine an instrument identity in the instrument library that matches the instrument image based on the normalized features and the instrument features of each instrument in the instrument library; wherein the instrument library is configured to store instrument features of multiple instruments, the instrument features including instrument ranges; and when the instrument image matches the corresponding instrument identity, the instrument features corresponding to the instrument identity are used as instrument features of the instrument type.

[0155] In a possible implementation, the detection module 303 is further used to: perform multi-pointer key point detection on the instrument image through the RepViT-yolov8-pose model to obtain multiple pointer key point coordinates of each pointer in the instrument image; wherein the multiple pointer key point coordinates of each pointer include the pointer root coordinates and the pointer tip coordinates of each pointer; wherein the RepViT-yolov8-pose model includes key point loss weights and label smoothing.

[0156] In a possible implementation, the instrument reading device further includes an acquisition module for acquiring the instrument image through a yolov8-repvit target detection model; wherein the backbone network of the yolov8-repvit target detection model is a RepViT lightweight Transformer.

[0157] In a possible implementation, the instrument reading device further includes a preprocessing module for predicting a rotation angle of the instrument image through an angle regression network based on RepViT; rotating the instrument image along the image center according to the rotation angle; and preprocessing the rotated instrument image.

[0158] In a possible implementation, the identification module 302 is specifically configured to: identify the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the preprocessed rotated instrument image through a key point model.

[0159] In a possible implementation manner, the detection module 303 is specifically configured to perform multi-pointer key point detection on the pre-processed and rotated instrument image to obtain coordinates of multiple pointer key points of each pointer in the instrument image.

[0160] In a possible implementation, the instrument reading device further includes a perspective transformation module, which is used to establish a polar coordinate system based on the distance between the target scale and the center of the instrument, and the target scale angle; map the polar coordinate system to a Cartesian coordinate system to obtain the converted target scale key point coordinates; determine a perspective transformation matrix based on the converted target scale key point coordinates and the target scale key point coordinates before the conversion; and perform perspective transformation on the rotated instrument image based on the perspective transformation matrix.

[0161] In a possible implementation manner, the reading determination module 304 is specifically used to determine the reading of the corresponding instrument according to the coordinates of multiple pointer key points after perspective transformation, the coordinates of target scale key points after perspective transformation, and the instrument range.

[0162] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the meter reading method described in the above method embodiment are executed.

[0163] The computer program product of the meter reading method provided in the embodiment of the present application includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the meter reading method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0164] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0166] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of more restrictions, the elements defined by the sentence "includes..." do not exclude the presence of other identical elements in the process, method, article or apparatus that includes the elements.

[0167] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0168] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for reading an instrument, characterized in that: include: Match the corresponding instrument features according to the instrument image; Identify the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the instrument image through the key point model; wherein the key point model is obtained by training the target scale key points and the instrument center key points of multiple types of instruments; Performing multi-pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image; The reading of the corresponding instrument is determined according to the coordinates of the multiple pointer key points corresponding to each pointer, the coordinates of the target scale key points and the instrument characteristics.

2. The method according to claim 1, characterized in that The matching of corresponding instrument features according to the instrument image includes: Obtaining normalized features of the instrument image through a preset model; and Determining an instrument identity in the instrument library that matches the instrument image according to the normalized feature and the instrument features of each instrument in the instrument library; wherein the instrument library is configured to store instrument features of a plurality of instruments, and the instrument features include an instrument range; When the instrument image matches the corresponding instrument identity mark, the instrument feature corresponding to the instrument identity mark is used as the instrument feature of the instrument type.

3. The method according to claim 2, characterized in that in, The preset model uses the ViT-Tiny as the backbone network; Center Loss is used as the loss function for training the preset model.

4. The method according to claim 1, characterized in that The performing multiple pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image includes: Using the RepViT-yolov8-pose model, multiple pointer key point detection is performed on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image; wherein the multiple pointer key point coordinates of each pointer include the pointer root coordinates and the pointer tip coordinates of each pointer; Among them, the RepViT-yolov8-pose model includes key point loss weights and label smoothing.

5. The method according to any one of claims 1 to 4, characterized in that: Before matching the corresponding instrument features according to the instrument image, the method further includes: Obtain the instrument image through the yolov8-repvit target detection model; Among them, the backbone network of the yolov8-repvit target detection model is the RepViT lightweight Transformer.

6. The method according to any one of claims 1 to 4, characterized in that: Before matching the corresponding instrument features according to the instrument image, the method further includes: Predicting the rotation angle of the instrument image through an angle regression network based on RepViT; Rotate the instrument image along the image center according to the rotation angle; Preprocess the rotated instrument image; The identifying, by using the key point model, the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the instrument image includes: Using the key point model, the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the preprocessed rotated instrument image are identified; The performing multiple pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image includes: The pre-processed and rotated instrument image is subjected to multi-pointer key point detection to obtain coordinates of multiple pointer key points of each pointer in the instrument image.

7. The method according to claim 6, characterized in that in, The instrument features include a target scale angle and an instrument range, wherein the target scale angle is the angle of the target scale relative to the starting time; Before determining the reading of the corresponding instrument according to the coordinates of the multiple pointer key points corresponding to each pointer, the target scale key points and the instrument features, the method further includes: Establishing a polar coordinate system based on the distance between the target scale and the center of the instrument and the target scale angle; Mapping the polar coordinate system to a Cartesian coordinate system to obtain the converted target scale key point coordinates; Determine the perspective transformation matrix according to the coordinates of the target scale key points after the transformation and the coordinates of the target scale key points before the transformation; Performing perspective transformation on the rotated instrument image based on the perspective transformation matrix; The step of determining the reading of the corresponding instrument according to the coordinates of the multiple pointer key points corresponding to each pointer, the target scale key points and the instrument features includes: The reading of the corresponding instrument is determined according to the coordinates of multiple pointer key points after perspective transformation, the coordinates of target scale key points after perspective transformation and the instrument range.

8. An instrument reading device, characterized in that: include: A type determination module, used to match corresponding instrument features according to the instrument image; A recognition module, used to recognize the target scale key point coordinates and the instrument center key point coordinates corresponding to the target scale in the instrument image through a key point model; wherein the key point model is obtained by training the target scale key points and the instrument center key points of multiple types of instruments; A detection module, used to perform multi-pointer key point detection on the instrument image to obtain multiple pointer key point coordinates of each pointer in the instrument image; The reading determination module is used to determine the reading of the corresponding instrument according to the coordinates of multiple pointer key points corresponding to each pointer, the coordinates of the target scale key points and the instrument characteristics.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of any method described in claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

Images