Pointer instrument reading method based on rotary YOLOv8 target detection network

Abstract

The invention discloses a pointer instrument reading method based on a rotating YOLOv8 target detection network. The pointer instrument reading method comprises the following steps: S1, representing the attitude of a rotating target; s2, rotation target loss calculation; s3, calculating a rotation IoU value; s4, identifying scale readings; through the above steps, the error between the predicted included angle between the pointer and the horizontal direction and the error between the manually marked included angle between the pointer and the horizontal direction are used as evaluation indexes, the method is equivalent to normalizing scales of different instruments, and a rotating rectangular frame is predicted on the basis of not greatly modifying an original network. And meanwhile, the model prediction speed is ensured, and the influence of the measuring range on reading errors is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of pointer instrument reading, and in particular to a pointer instrument reading method based on a rotating YOLOv8 target detection network. Background Art

[0002] The so-called rotation target detection is to change the bounding rectangle of the predicted target to the minimum bounding rectangle of the predicted target based on the horizontal target detection algorithm, so as to obtain the length, width and angle information of the target more accurately and make the feature information output by the model richer. There are three main ways to represent the rotation target:

[0003] Method 1: Directly predict the coordinate values ​​of the four vertices of the rotated rectangle, (x1, y1), (x2, y2), (x3, y3), (x4, y4);

[0004] In the above method 1, based on the non-rotated rectangle, a quadrilateral is located to represent an object by learning the offset of four points on the non-rotated rectangle. The marking method used by this model is as follows Figure 7 As shown, when predicting a target, a horizontal rectangle is first predicted, then 4 offsets are predicted, and then the coordinates of the rotated rectangle are calculated. The main purpose of designing a rotating target detection network in this application is to predict the pointer direction. This method has high complexity, so it is necessary to simplify the problem and reduce the difficulty.

[0005] Method 2: Predict the two vertex coordinates and an offset of a long side of the target;

[0006] In the above method 2, the rotated rectangle is defined by the coordinates of the first two vertices (p, θ1) and (p, θ2) starting from the upper left corner in the clockwise direction, plus the regression pole (x, y), a total of 5-dimensional information to define a rotated rectangle. This can avoid the instability caused by directly predicting the angle information of the rotated rectangle. The rotated rectangle marking method used by this model is as follows Figure 8 As shown, this application seeks to make minor adjustments to the original detection model, which has undergone significant changes, so this solution is not considered.

[0007] Method 3: Construct a rotation anchor, which contains information such as width, height, and rotation angle, and use the rotation anchor to fit the actual minimum area circumscribed rectangle;

[0008] Method 3: Rotated rectangle definition: (x, y) is the coordinate of the center point of the rotated rectangle, (w, h) are the long and short sides of the rectangle respectively, and the angle θ is the angle between the positive direction of the X-axis and the first long side of the rectangle. In order to reduce the difficulty of model fitting, RRPN increases the number of anchors (3 scales, 3 aspect ratios) corresponding to one grid on the feature map to 54 (3 scales, 3 aspect ratios, 6 angles). The schematic diagram of the anchor used by RRPN is as follows: Fig. 9 As shown in the figure, although increasing the number of anchors can reduce the difficulty of model fitting, it also increases the time consumption of model detection.

[0009] From the above analysis, it can be seen that method 1 and method 2 make significant changes to the original target detection network, the network is more complex, and in order to ensure the accuracy of learning, it is generally necessary to modify the two-stage network, and the model prediction speed is slow. Method 3 increases the number of anchors by adding rotation anchor information. Although it reduces the difficulty of model learning to a certain extent, it also seriously increases the prediction time of the model.

[0010] Therefore, a pointer instrument reading method based on rotating YOLOv8 target detection network is proposed. Summary of the invention

[0011] The purpose of the present invention is to solve the shortcomings of the prior art mentioned in the background technology, and to propose a pointer instrument reading method based on a rotating YOLOv8 target detection network.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions:

[0013] A pointer instrument reading method based on the rotating YOLOv8 target detection network is proposed. Taking YOLOv8 as the benchmark network, R-YOLOv8 is proposed by defining the representation of the rotated rectangle, designing a suitable loss function and a rotation IoU threshold calculation method, including the following steps:

[0014] S1, attitude representation of rotating target;

[0015] S11, determine the definition method of the rotated rectangle: (x, y, w, h, θ), where (x, y) is the coordinate of the center point of the rotated rectangle, w is the short side of the rectangle, h is the long side of the rectangle, and the rotation angle θ is the angle between the long side h and the horizontal direction;

[0016] S12. Measure the impact of the loss, convert the angle regression problem into a classification problem, solve the PoA problem that occurs during the regression process, and use the maximum accuracy loss and average loss to measure the impact of the loss;

[0017] S2, calculation of rotation target loss;

[0018] The loss function of the rotation angle is shown in 3-9;

[0019]

[0020] S3, calculation of rotation IoU value;

[0021] Calculate the IOU value of every two rotated boxes, and distinguish the positive and negative samples of all rotated boxes and real boxes predicted by each grid;

[0022] S4, identify the scale indication;

[0023] The location and recognition of dial scales belong to the recognition of pointer instrument readings. Based on this, the scale numbers with similar spatial distances are merged through the spatial clustering algorithm to obtain the instrument scale value with semantics.

[0024] The merging process is as follows: S41, first obtain the detected digital frame result B, B contains the detection results of n digital frames, and the format of each detection result is [x, y, w, h, θ, c];

[0025] S42, traverse all the boxes in B and find the longest long side h max , set the distance threshold L to 2.5×h max ;

[0026] S43. Calculate the distance between the digital boxes in B.

[0027] As a further feature of the method of the present invention, the value range of S111 and θ is [-90, 90).

[0028] As a further step in the method of the present invention, S121, a classification method is selected to predict the pointer rotation angle. When predicting the pointer direction, if the predicted angle is close to the true angle but not the same as the true angle, such as the true angle is 1° and the predicted result is 3°, then this prediction result is also acceptable.

[0029] As a further step in the method of the present invention, S1211, a circular smooth label is cited to divide the loss size of different categories, wherein the circular smooth label measures the distance between the predicted label and the true label by setting a window function.

[0030] As a further step in the method of the present invention, S31 mainly calculates the intersection-and-union ratio of two rotated rectangles.

[0031] As a further step in the method of the present invention, S311, firstly calculate all intersection points of two rotated rectangles;

[0032] S312, find the point with the smallest y coordinate under the premise of the smallest x coordinate as the vertex;

[0033] S313, connecting the vertex found in step S312 with each of the remaining vertices, and dividing the intersecting area into a plurality of triangles;

[0034] S314. Finally, the areas of all triangles are calculated and summed to obtain the IoU threshold of the two rotated rectangles.

[0035] As a further step in the method of the present invention, S411, wherein x, y represent the center coordinates of the box, w, h represent the short side and long side of the box, θ represents the angle between the long side h of the box and the x-axis, and c represents the category of the digital box, that is, the corresponding number.

[0036] As a further step in the method of the present invention, S431, if the distance between the i-th box and the j-th box is less than the threshold L, the two boxes are merged, the merged box is added to B, and the i-th box and the j-th box are deleted, and the process of merging digital boxes and updating B is repeated until the distance between all digital boxes in B is greater than L, and the combined digital box result B is obtained after the merging is completed.

[0037] Beneficial Effects of the Invention

[0038] This application uses the error between the predicted angle between the pointer and the horizontal direction and the angle between the manually marked pointer and the horizontal direction as an evaluation indicator. Using this method is equivalent to normalizing the scales of different instruments, predicting the rotating rectangular box without significantly modifying the original network, while ensuring the speed of model prediction and avoiding the influence of the range on the reading error. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a rotation rectangle definition in the method of the present invention;

[0040] Figure 2 Schematic diagram of the angles of the rotating frame at three different positions in the method of the present invention;

[0041] Figure 3 A schematic diagram of a circular smooth label in the method of the present invention;

[0042] Figure 4 It is a regression schematic diagram of a rotating rectangular frame in the method of the present invention;

[0043] Figure 5 Schematic diagram of the calculation method of the IOU value of a regular rectangle in the method of the present invention;

[0044] Figure 6 Schematic diagram of calculation of rotation IOU value in the method of the present invention;

[0045] Figure 7 It is a schematic diagram of the rotating rectangle marking method in the existing method;

[0046] Figure 8 Schematic diagram of polar coordinate rotation rectangle in existing methods;

[0047] Fig. 9 Schematic diagram of RRPN rotation anchor in existing methods. DETAILED DESCRIPTION

[0048] The content of the present invention can be more easily understood by selecting the following detailed description of the preferred implementation method of the present invention and the embodiments included. Unless otherwise specified, all technical and scientific terms used in this application have the same meanings as those of ordinary skill in the art to which the present invention belongs. When there is a conflict, the definition in this specification shall prevail.

[0049] As used herein, the term "prepared from" is synonymous with "comprising". As used herein, the terms "comprising", "including", "having", "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, process, method, article, or apparatus comprising the listed elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such composition, process, method, article, or apparatus.

[0050] The conjunction "consisting of excludes any unspecified element, step, or component. If used in a claim, this phrase renders the claim closed-ended so that it does not include materials other than those described, except for conventional impurities associated therewith. When the phrase "consisting of" appears in a clause of the body of a claim rather than immediately following the subject matter, it limits only the elements described in that clause; other elements are not excluded from the claim as a whole.

[0051] When amount, concentration or other value or parameter is represented by range, preferred range or a series of upper preferred value and lower preferred value, this should be understood as specifically disclosing all ranges formed by any pairing of any range upper limit or preferred value and any range lower limit or preferred value, regardless of whether the range is disclosed separately. For example, when disclosing range "1 to 5", described range should be interpreted as including range "1 to 4", "1 to 3", "1 to 2", "1 to 2 and 4 to 5", "1 to 3 and 5" etc. When numerical range is described in the application, unless otherwise stated, the range is intended to include its end value and all integers and fractions within the range.

[0052] Singular forms include plural references unless the context clearly indicates otherwise. "Optional" or "either" means that the subsequently described event or incident may or may not occur, and that the description includes instances where the event occurs and instances where it does not.

[0053] Approximate terms in the specification and claims are used to modify quantities, indicating that the present invention is not limited to the specific quantity, but also includes acceptable and modified parts close to the quantity without causing changes in the relevant basic functions. Accordingly, the use of "about", "approximately", etc. to modify a numerical value means that the present invention is not limited to the exact numerical value. In some examples, the approximate terms may correspond to the accuracy of the instrument for measuring the numerical value. In the specification and claims of this application, range limitations can be combined and / or interchanged, and if not otherwise stated, these ranges include all subranges contained therein.

[0054] In addition, the indefinite articles "a" and "an" before the elements or components of the present invention have no restrictions on the quantity requirements (i.e. the number of occurrences) of the elements or components. Therefore, "a" or "an" should be interpreted as including one or at least one, and the elements or components in the singular form also include the plural form, unless the number is obviously intended to be in the singular form.

[0055] In the description of the present invention, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0056] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0057] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "mounted / connected", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] The technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0059] In order to predict the rotated rectangle without significantly modifying the original network and at the same time ensure the speed of model prediction, this application adopts a rotated rectangle definition method similar to method three, as follows.

[0060] Reference Figure 1-6 , a pointer instrument reading method based on the rotating YOLOv8 object detection network, takes YOLOv8 as the benchmark network, and proposes R-YOLOv8 by defining the representation of rotated rectangles, designing appropriate loss functions and rotating IoU value calculation methods;

[0061] The following steps are involved:

[0062] S1, attitude representation of rotating target;

[0063] S11. Determine the definition of the rotated rectangle: (x, y, w, h, θ), where (x, y) is the coordinate of the center point of the rotated rectangle, w is the short side of the rectangle, h is the long side of the rectangle, and the rotation angle θ is the angle between the long side h and the horizontal direction. For details, see Figure 1 ;

[0064] Depend on Figure 1 It can be seen that the value range of θ is [-90,90). The rotated rectangle represented by method 3 has angle periodicity problem when regressing the angle parameter, such as Figure 2 As shown, the angle between box ② and the axis is 89°. Box ② rotates 1° counterclockwise to become the position of box ①, and the angle is -90°. Box ② rotates 1° clockwise to become the position of box ③, and the angle is 88°. The actual rotation angle from box ③ to box ① is 2°, but the defined rotation angle differs by about one cycle. During the angle parameter regression process in the network, the loss function will produce sharp oscillations, and the network will be difficult to train.

[0065] S12. Measure the impact of the loss;

[0066] In order to solve the PoA problem in the regression process, the angle regression problem is converted into a classification problem. Since the continuous problem is discretized, the loss of accuracy is inevitable. When representing one class, there will be a loss of accuracy. In order to calculate the impact of the accuracy loss, the maximum accuracy loss and the average loss (subject to uniform distribution) are used here to measure the impact of the loss.

[0067] Max(loss)=ω / 2......(3-1)

[0068]

[0069] When 1° is classified into one category (ω=1), it can be seen from formulas (3-1) and (3-2) that the maximum and average precision losses are 0.5 and 0.25 respectively. This precision loss is within the negligible error range for the angle direction prediction task. Therefore, the classification method can be used to predict the pointer rotation angle. When predicting the pointer direction, if the predicted angle is close to the true angle but not the same as the true angle, for example, the true angle is 1° and the predicted result is 3°, then this prediction result is also acceptable. Therefore, the annular smoothing label ( Figure 3 ), to classify the loss size of different categories;

[0070] The circular smoothing label measures the distance between the predicted label and the true label by setting the window function. The formula is as follows (3-3):

[0071]

[0072] g(x) in formula (3-3) is a window function with periodicity and symmetry. The window function selected in this application is the Gaussian function with a window radius of 3. The final expression is shown in formula (3-4), where a, b, c are all constant terms; x is the input angle information; θ is the angle value to be rotated.

[0073]

[0074] S2. Rotation target loss calculation

[0075] The fitting method of the rotating rectangular frame is as follows Figure 4 As shown in the figure, the purple dotted line represents the best anchor (Pw, Ph) corresponding to the grid on the feature map, that is, the anchor with the largest intersection ratio with the real rectangular box among the preset anchors. The red rectangle represents the real rotated rectangular box (bw, bh). The grid coordinates are (Cx, Cy), and the offset of the center point is (σ(tx), σ(ty)). For the YOLOv8 target detection algorithm, the purpose of bounding box regression is to learn the mapping and offset (σ(tx), σ(ty)) from (Pw, Ph) to (bw, bh). Therefore, the loss function of the YOLOv8 target detection algorithm consists of three parts as shown in formula (3-5), L obj , L box , L cls Represent confidence loss, bounding box regression loss and category loss respectively;

[0076] Loss = L obj +L box +L cls ..........3-5

[0077] YOLOv8 uses GIoU as the loss function for bounding box regression. Both classification loss and confidence loss use binary cross entropy loss. The calculation formulas for GIoU and GIoUloss are as follows:

[0078]

[0079] GIoUloss=1-GIoU..........3-7

[0080] Where A is the predicted box, B is the real box, and C represents the minimum convex closed box containing A and B;

[0081] The total loss calculation formula of the model is shown below (3-8). The first term represents the bounding box regression loss, the second and third terms represent the confidence prediction loss, and the fourth term represents the category prediction loss:

[0082]

[0083] In the detection of rotated objects, the purpose of bounding box regression is to learn the mapping from (Pw, Ph) to (bw, bh) and the offset (σ(tx), σ(ty)) and the rotation angle θ. Therefore, this application designs the loss function shown in formula (3-9), which adds the loss of rotation angle on the basis of horizontal bounding box loss.

[0084]

[0085]

[0086] S3. Rotation IoU value calculation

[0087] During model training, in order to distinguish the positive and negative samples of all the rotated boxes and the real boxes predicted by each grid, the IOU value of every two rotated boxes needs to be calculated. The IOU value is usually calculated as follows: Figure 5 As shown, there are many forms of intersection between two rotated rectangles, and it is relatively difficult to calculate the intersection ratio. Figure 6 The method shown is used to calculate the IoU ratio of two rotated rectangles. First, all the intersection points of the two rotated rectangles are calculated. Then, the point with the smallest y coordinate under the premise of the smallest x coordinate is found as the vertex. Then, the vertex is connected to each of the remaining vertices to divide the intersection area into several triangles. Finally, the areas of all triangles are calculated and summed to obtain the IoU threshold of the two rotated rectangles.

[0088] S4. Identify the scale indication according to the above algorithm steps

[0089] S41. The location and recognition of dial scales belong to the recognition of pointer instrument readings. Based on this, the scale numbers with similar spatial distances are merged through a spatial clustering algorithm to obtain instrument scale values ​​with semantics.

[0090] S411, the merging process is as follows:

[0091] First, get the detected digital frame result B, which contains the detection results of n digital frames. The format of each detection result is [x, y, w, h, θ, c], where x and y represent the center coordinates of the frame, w and h represent the short and long sides of the frame, θ represents the angle between the long side h of the frame and the x-axis, and c represents the category of the digital frame, i.e. the corresponding number. Then traverse all the frames in B and find the longest long side h. max , set the distance threshold L to 2.5×h max After multiple tests, this application found that the distance threshold is set to 2.5×h max The effect is better when , and then calculate the distance between the digital boxes in B. If the distance between the i-th box and the j-th box is less than the threshold L, the two boxes are merged, the merged box is added to B, and the i-th box and the j-th box are deleted. Repeat the process of merging digital boxes and updating B until the distance between all digital boxes in B is greater than L. After the merger is completed, the combined digital box result B is obtained;

[0092] The pseudo code corresponding to the above merging process is as follows:

[0093] Input number box set B{b1,b2,b3......b n}, distance threshold L, state flag F, distance function len(·), maximum value function max(·), digital box combination function combine(·), collection box update function update(·);

[0094] Output combination number box B.

[0095] BEGIN

[0096] L←O,,F←O

[0097] L←2.5×h max ,h max ←max(B,h)

[0098] WHILEF≠O,

[0099] F←O,

[0100] FOR i ,b j ∈B AND b i ≠b i DO,

[0101] IF len(b i ,b j )<L THEN b ij←combine(b i ,b j )

[0102] update(B,b ij ,b i ,b j )

[0103] F←1

[0104] BREAK

[0105] END

[0106] RETURN

[0107] END

[0108] Summary: The pointer instrument reading method based on rotating target detection proposed in this application is based on the rotating target detection network R-YOLOv8. Specifically, by comparing three different rotated rectangle representations, it is determined to use the long side representation to represent a rotated rectangle. By analyzing the regression method of the rotated rectangle, the loss function of R-YOLOv8 is proposed. By analyzing the intersection form of the rotated rectangle, a rotation IoU threshold calculation method is proposed. Then, for the problem of dial scale recognition, the dial scale recognition method proposed in this paper is introduced, and the performance of the instrument reading recognition algorithm proposed in this paper is analyzed. In order to solve the problem of shortage of pointer instrument data, a data enhancement method is proposed to avoid errors and enhance the accuracy of readings.

[0109] The examples involved in this application are merely illustrative and are used to explain some features of the method of the present invention. The attached claims are intended to require the widest possible range that can be imagined, and the embodiments presented in this application are only descriptions of the selected implementation methods according to the combination of all possible embodiments. Therefore, it is the applicant's intention that the attached claims are not limited by the selection of examples that illustrate the features of the present invention. Some numerical ranges used in the claims also include sub-ranges within them, and changes in these ranges should also be interpreted as being covered by the attached claims where possible.

Claims

1. A pointer instrument reading method based on a rotating YOLOv8 target detection network, characterized in that: The following steps are involved: S1, attitude representation of rotating target; S11, determine the definition method of the rotated rectangle: (x, y, w, h, θ), where (x, y) is the coordinate of the center point of the rotated rectangle, w is the short side of the rectangle, h is the long side of the rectangle, and the rotation angle θ is the angle between the long side h and the horizontal direction; S12. Measure the impact of the loss, convert the angle regression problem into a classification problem, solve the PoA problem that occurs during the regression process, and use the maximum accuracy loss and average loss to measure the impact of the loss; S2, calculation of rotation target loss; The loss function of the rotation angle is shown in 3-9; S3, calculation of rotation IoU value; Calculate the IOU value of every two rotated boxes, and distinguish the positive and negative samples of all rotated boxes and real boxes predicted by each grid; S4, identify the scale indication; The location and recognition of dial scales belong to the recognition of pointer instrument readings. Based on this, the scale numbers with similar spatial distances are merged through the spatial clustering algorithm to obtain the instrument scale value with semantics. The merging process is as follows: S41, first obtain the detected digital frame result B, B contains the detection results of n digital frames, and the format of each detection result is [x, y, w, h, θ, c]; S42, traverse all the boxes in B and find the longest long side h max , set the distance threshold L to 2.5×h max ; S43. Calculate the distance between the digital boxes in B.

2. A pointer instrument reading method based on a rotating YOLOv8 target detection network according to claim 1, characterized in that: The value range of S111, θ is [-90, 90).

3. A pointer instrument reading method based on a rotating YOLOv8 target detection network according to claim 1, characterized in that: S121. Use a classification method to predict the pointer rotation angle. When predicting the pointer direction, if the predicted angle is close to but not the same as the true angle, such as the true angle is 1° and the predicted result is 3°, then this prediction result is also acceptable.

4. A pointer instrument reading method based on a rotating YOLOv8 target detection network according to claim 1, characterized in that: S1211. A circular smooth label is used to divide the loss size of different categories, where the circular smooth label measures the distance between the predicted label and the true label by setting a window function.

5. A pointer instrument reading method based on a rotating YOLOv8 target detection network according to claim 1, characterized in that, S31, mainly calculates the intersection ratio of two rotated rectangles.

6. A pointer instrument reading method based on a rotating YOLOv8 target detection network according to claim 1, characterized in that: S311, first calculate all intersection points of two rotated rectangles; S312, find the point with the smallest y coordinate under the premise of the smallest x coordinate as the vertex; S313, connecting the vertex found in step S312 with each of the remaining vertices, and dividing the intersecting area into a plurality of triangles; S314. Finally, the areas of all triangles are calculated and summed to obtain the IoU values ​​of the two rotated rectangles.

7. A pointer instrument reading method based on a rotating YOLOv8 target detection network according to claim 1, characterized in that: S411, wherein x, y represent the center coordinates of the box, w, h represent the short side and long side of the box, θ represents the angle between the long side h of the box and the x-axis, and c represents the category of the digital box, i.e., the corresponding number.

8. A pointer instrument reading method based on a rotating YOLOv8 target detection network according to claim 1, characterized in that: S431. If the distance between the i-th box and the j-th box is less than the threshold L, the two boxes are merged, the merged box is added to B, and the i-th box and the j-th box are deleted. The process of merging digital boxes and updating B is repeated until the distance between all digital boxes in B is greater than L. After the merging is completed, the combined digital box result B is obtained.

Images