HRNet-based pointer instrument identification method and device and medium

Abstract

The invention discloses a pointer type instrument identification method and device based on HRNet and a medium, and relates to the technical field of rail transit intelligence. The method comprises the following steps: based on an instrument data set, obtaining an instrument identification model through HRNet network training; and obtaining a real-time image of an instrument, cutting the real-time image of the instrument according to a preset region of interest, and inputting the cut real-time image into the instrument identification model to obtain an instrument value. The method provided by the invention solves the problems that an existing monitoring video lacks automatic analysis, important information contained in the video cannot be recognized in time, abnormal reading of a pointer type instrument cannot be found in time, and major potential safety hazards exist.

Description

Technical Field

[0001] The present invention relates to the field of rail transit intelligent technology, and more specifically, to a pointer instrument recognition method, device and medium based on HRNet. Background Art

[0002] The pump room and many other facilities and equipment in the high-speed railway station area provide flood control and drainage, fire-fighting water, and production and living water services for the lines, stations and production units. They are directly related to line safety, driving safety and the quality of station passenger services, and the responsibility is heavy.

[0003] The current water supply and drainage operation and maintenance of high-speed railways is mainly based on extensive human input, relying on a large number of manpower, manual duty in pump rooms, regular maintenance by manpower, regular inspection of lines by manpower, and emergency repairs in case of emergencies. It is necessary to deploy duty personnel at each station to manually monitor the operation status of equipment, and the maintenance service workload is very large. The existing pump rooms are basically equipped with high-definition cameras to monitor the operating environment, but the massive real-time monitoring videos they provide lack automatic analysis, and cannot timely identify important information contained in the videos, and cannot promptly detect abnormal readings of pointer instruments, which poses a major safety hazard. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a pointer instrument recognition method, device and medium based on HRNet to solve the problems that the existing surveillance video lacks automatic analysis, cannot timely identify the important information contained in the video, cannot timely discover the abnormal readings of the pointer instrument, and has major safety hazards.

[0005] In a first aspect, the present invention provides a pointer instrument recognition method based on HRNet, the method comprising:

[0006] Based on the instrument data set, the instrument recognition model is obtained through HRNet network training;

[0007] The real-time image of the instrument is acquired, and the real-time image of the instrument is cropped according to a preset region of interest and then input into the instrument recognition model to obtain the instrument value.

[0008] Furthermore, the instrument data set includes an instrument image and a corresponding pointer scale label; the pointer scale label includes a pointer base point coordinate, a pointer tail point coordinate, and a start scale coordinate to an end scale coordinate.

[0009] Furthermore, based on the instrument data set, the instrument recognition model is obtained through HRNet network training, including:

[0010] The instrument images and corresponding pointer scale labels in the instrument dataset are imported into the HRNet network for training to obtain the instrument recognition model.

[0011] Furthermore, the HRNet network includes multiple convolutional layers; wherein, the output results of at least two convolutional layers are downsampled using a maximum pooling layer to reduce the size of the feature map by half.

[0012] Furthermore, the HRNet network includes 20 convolutional layers; wherein:

[0013] The first and second layers are convolutional layers with a kernel of 3×3, a stride of 2, and 64 output channels;

[0014] The third layer is a maximum pooling layer with a kernel of 2×2 and a stride of 2;

[0015] From the 4th to the 7th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 64. Each convolutional layer is followed by normalization and activation;

[0016] From the 8th to the 11th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 128;

[0017] From the 12th to the 15th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 256;

[0018] From the 16th to the 19th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 512;

[0019] The results of the 11th, 15th, and 19th stages are downsampled using a 2×2 max pooling layer to halve the feature map size; the output of each stage is fused with the output of the previous stage;

[0020] The 20th layer is a fully connected layer with a kernel of 1×1 and an output channel number of 3×c, where c represents the number of key points.

[0021] Furthermore, a real-time image of the instrument is obtained, and the real-time image of the instrument is cropped according to a preset region of interest and then input into the instrument recognition model to obtain the instrument value, including:

[0022] Collect real-time images, crop the images according to the preset regions of interest, and feed them into the instrument recognition model;

[0023] The output of the instrument recognition model is obtained to obtain n vectors; each vector represents the detection result of a dashboard; each vector is composed of [x -1 ,y -1 ,x0,y0,x1,y1,x2,y2,…,x m ,y m], which represent the horizontal and vertical coordinates of the pointer base point, the horizontal and vertical coordinates of the pointer tail point, and the horizontal and vertical coordinates of the first large scale to the mth large scale; extract the pointer base point R and tail point T, whose coordinates are [x -1 ,y -1 ], [x0, y0]; extract the scale coordinate P i , the coordinates are [x i ,y i ], where i = 1, 2, ... m, m is the number of scales;

[0024] Calculate the scale index k that satisfies arg(RP k )<arg(RT)<arg(RP k+1 );RPk、RT、RPk+1 are the vectors from the pointer base point to the kth scale, the pointer tail point, and the k+1th scale respectively, arg represents the angle between the vector and the vertical axis; the proportional value a is calculated according to the angle between the vector corresponding to the scale index k and the vertical axis; the instrument value is calculated according to the proportional value a and the preset range L.

[0025] Furthermore, according to the angle between the vector corresponding to the scale index k and the vertical axis, the scale value a is calculated by the following formula:

[0026]

[0027] Furthermore, according to the ratio value a and the preset range L, the instrument value is calculated by the following formula:

[0028]

[0029] In a second aspect, the present invention provides a pointer instrument recognition device based on HRNet, the device comprising:

[0030] The model training unit is configured to obtain an instrument recognition model through HRNet network training based on the instrument data set;

[0031] The value recognition unit is configured to obtain a real-time image of the instrument, crop the real-time image of the instrument according to a preset region of interest, and then input the cropped image into the instrument recognition model to obtain the instrument value.

[0032] In a third aspect, the present invention provides a readable storage medium, wherein the readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method as described above.

[0033] The present invention has at least the following beneficial effects:

[0034] The present invention can identify the readings of pointer instruments in instrument images in real time based on instrument images captured by high-definition cameras, so that the readings of pointer instruments can be obtained in time, and based on set thresholds, once the thresholds are exceeded, alarm information is reported immediately to improve safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of an application scenario according to the prior art is shown.

[0036] Figure 2 A flow chart of a pointer instrument recognition method based on HRNet according to an embodiment of the present invention is shown.

[0037] Figure 3 A diagram showing the output result of the instrument identification model according to an embodiment of the present invention is shown.

[0038] Figure 4 A structural diagram of a pointer instrument recognition device based on HRNet according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments, but are not intended to limit the present invention. For the various steps described herein, if there is no necessity for a causal relationship between each other, the order in which they are described as examples herein should not be regarded as a limitation, and those skilled in the art should know that they can be adjusted in order, as long as the logic between them is not destroyed, resulting in the inability to implement the entire process.

[0040] Figure 1 A schematic diagram of an application scenario according to the prior art is shown. Figure 1 As shown, the application scenario includes a camera component 101 and a data processing terminal 102, wherein the camera component 101 can select an existing high-definition camera, which is deployed in the pump room of the high-speed railway to monitor the operating environment, and the acquired video data stream is fed to the data processing terminal 102 in real time, and the data processing terminal 102 is configured with an instrument recognition model. The data processing terminal 102 extracts the real-time image of the instrument based on the video data stream fed in real time by the camera component 101, crops the real-time image of the instrument according to the preset area of ​​interest, and inputs it into the instrument recognition model to obtain the instrument value, so as to perform real-time reading of the pointer instrument. Once the threshold is exceeded, the alarm information is immediately reported, so that the important information contained in the video can be identified in time, and abnormal readings of the pointer instrument can be discovered in time, thereby eliminating the major safety hazards that may exist.

[0041] The embodiment of the present invention provides a pointer instrument recognition method based on HRNet. The main purpose of the method is to construct an instrument recognition model, and the constructed recognition model can be configured in Figure 1 In the data processing end 102 in the illustrated application scenario, the data processing end 102 recognizes the real-time image of the instrument based on the instrument recognition model to obtain the instrument value.

[0042] Specifically, if Figure 2 FIG. 1 is a flow chart of a pointer instrument recognition method based on HRNet. The pointer instrument recognition method based on HRNet includes steps S10 to S20, which are described in detail as follows.

[0043] S10: Based on the instrument dataset, the instrument recognition model is obtained through HRNet network training.

[0044] In some embodiments, the instrument data set includes an instrument image and a corresponding pointer scale label; the pointer scale label includes a pointer base point coordinate, a pointer tail point coordinate, and a start scale coordinate to an end scale coordinate.

[0045] In some embodiments, the instrument recognition model may be constructed by the following method:

[0046] The instrument data set includes pointer scale labels, which include pointer base point coordinates, pointer tail point coordinates, start scale coordinates, and end scale coordinates. The data set labels are imported into the HRNet network for training to obtain an instrument recognition model. The input image size of the HRNet network is 3×256×192, and its specific structure includes:

[0047] The first and second layers are convolutional layers with a kernel of 3×3, a stride of 2, and 64 output channels;

[0048] The third layer is a maximum pooling layer with a kernel of 2×2 and a stride of 2;

[0049] From the 4th to the 7th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 64. Each convolutional layer is followed by normalization and activation;

[0050] From the 8th to the 11th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 128;

[0051] From the 12th to the 15th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 256;

[0052] From the 16th to the 19th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 512;

[0053] The results of the 11th, 15th, and 19th stages are downsampled using a 2×2 max pooling layer to halve the feature map size; the output of each stage is fused with the output of the previous stage;

[0054] The 20th layer is a fully connected layer with a kernel of 1×1 and an output channel number of 3×c, where c represents the number of key points.

[0055] S20: Acquire a real-time image of the instrument, crop the real-time image of the instrument according to a preset region of interest, and input the cropped image into the instrument recognition model to obtain an instrument value.

[0056] In some embodiments, step S20 may be specifically implemented by the following steps:

[0057] S21: Collect real-time images, crop the images according to the preset regions of interest, and send them to the instrument recognition model.

[0058] S22: Obtain the output of the instrument recognition model to obtain n vectors; each vector represents the detection result of a dashboard; each vector is composed of [x -1 ,y -1 ,x0,y0,x1,y1,x2,y2,…,x m ,y m ], which represent the horizontal and vertical coordinates of the pointer base point, the horizontal and vertical coordinates of the pointer tail point, and the horizontal and vertical coordinates of the first large scale to the mth large scale; extract the pointer base point R and tail point T, whose coordinates are [x -1 ,y -1 ], [x0, y0]; extract the scale coordinate P i , the coordinates are [x i ,y i ], where i = 1, 2, … m, and m is the number of scales.

[0059] S23: Calculate the scale index k, satisfying arg(RP k )<arg(RT)<arg(RP k+1 ); RPk, RT, RP k+1 They are the vectors from the pointer base point to the kth scale, the pointer tail point, and the k+1th scale respectively. arg represents the angle between the vector and the vertical axis. The proportional value a is calculated based on the angle between the vector corresponding to the scale index k and the vertical axis. The instrument value is calculated based on the proportional value a and the preset range L.

[0060] In this embodiment, the ratio value a is calculated according to the angle between the vector corresponding to the scale index k and the vertical axis by the following formula:

[0061]

[0062] According to the ratio value a and the preset range L, the instrument value is calculated by the following formula:

[0063]

[0064] To further illustrate the feasibility of the present invention, a specific example is now used to illustrate the pointer instrument recognition method based on HRNet.

[0065] Specifically, the instrument data set in step S10 is a pointer scale label, including pointer base point coordinates, pointer tail point coordinates, starting scale coordinates to ending scale coordinates, a total of 7 coordinates, each coordinate includes a horizontal coordinate value and a vertical coordinate value; the number of output channels of the 20th layer included in the HRNet network structure is 3×14.

[0066] Specifically, step S21 uses the TensorRT graphics card inference calculation framework to perform model inference calculation. The output result of the instrument recognition model is as follows: Figure 3 shown.

[0067] Specifically, the instrument recognition model in step S22 outputs a vector; the coordinates of the pointer base point are [296, 263], and the coordinates of the pointer tail point are [174, 305]; the coordinates of the five scales are [161, 312], [207, 168], [342, 145], [413, 262], and [363, 356], respectively.

[0068] Specifically, in step S23, the scale index k is 1; the calculated ratio value a is 0.027; the preset range is 1.5, and the instrument value v=1.5 / (5-1)×(1-1+0.027)=0.01 is obtained.

[0069] The embodiment of the present invention also provides a pointer instrument recognition device based on HRNet, such as Figure 4 As shown, the device comprises:

[0070] The model training unit 401 is configured to obtain an instrument recognition model through HRNet network training based on the instrument data set;

[0071] The value recognition unit 402 is configured to obtain a real-time image of the instrument, crop the real-time image of the instrument according to a preset region of interest, and input the cropped image into the instrument recognition model to obtain the instrument value.

[0072] In some embodiments, the instrument data set includes an instrument image and a corresponding pointer scale label; the pointer scale label includes a pointer base point coordinate, a pointer tail point coordinate, and a start scale coordinate to an end scale coordinate.

[0073] In some embodiments, the model training unit is further configured to:

[0074] The instrument images and corresponding pointer scale labels in the instrument dataset are imported into the HRNet network for training to obtain the instrument recognition model.

[0075] In some embodiments, the HRNet network includes multiple convolutional layers; wherein the output results of at least two convolutional layers are downsampled using a maximum pooling layer to halve the size of the feature map.

[0076] In some embodiments, the HRNet network includes 20 convolutional layers; wherein:

[0077] The first and second layers are convolutional layers with a kernel of 3×3, a stride of 2, and 64 output channels;

[0078] The third layer is a maximum pooling layer with a kernel of 2×2 and a stride of 2;

[0079] From the 4th to the 7th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 64. Each convolutional layer is followed by normalization and activation;

[0080] From the 8th to the 11th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 128;

[0081] From the 12th to the 15th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 256;

[0082] From the 16th to the 19th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 512;

[0083] The results of the 11th, 15th, and 19th stages are downsampled using a 2×2 max pooling layer to halve the feature map size; the output of each stage is fused with the output of the previous stage;

[0084] The 20th layer is a fully connected layer with a kernel of 1×1 and an output channel number of 3×c, where c represents the number of key points.

[0085] In some embodiments, the value identification unit is further configured to:

[0086] Collect real-time images, crop the images according to the preset regions of interest, and feed them into the instrument recognition model;

[0087] The output of the instrument recognition model is obtained to obtain n vectors; each vector represents the detection result of a dashboard; each vector is composed of [x -1 ,y -1 ,x0,y0,x1,y1,x2,y2,…,x m ,ym ], which represent the horizontal and vertical coordinates of the pointer base point, the horizontal and vertical coordinates of the pointer tail point, and the horizontal and vertical coordinates of the first large scale to the mth large scale; extract the pointer base point R and tail point T, whose coordinates are [x -1 ,y -1 ], [x0, y0]; extract the scale coordinate P i , the coordinates are [x i ,y i ], where i = 1, 2, ... m, m is the number of scales;

[0088] Calculate the scale index k that satisfies arg(RP k )<arg(RT)<arg(RP k+1 );RPk、RT、RPk+1 are the vectors from the pointer base point to the kth scale, the pointer tail point, and the k+1th scale respectively, arg represents the angle between the vector and the vertical axis; the proportional value a is calculated according to the angle between the vector corresponding to the scale index k and the vertical axis; the instrument value is calculated according to the proportional value a and the preset range L.

[0089] In some embodiments, the numerical value identification unit is further configured to calculate the ratio value a according to the angle between the vector corresponding to the scale index k and the vertical axis by the following formula:

[0090]

[0091] In some embodiments, the value recognition unit is further configured to calculate the instrument value according to the ratio value a and the preset range L by the following formula:

[0092]

[0093] It should be noted that the structures of the various HRNet-based pointer meter recognition devices described in this embodiment belong to the same technical concept as the previously described HRNet-based pointer meter recognition method, and achieve the same beneficial effects through the same principles, which will not be repeated here.

[0094] An embodiment of the present invention further provides a readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the method described in any of the above embodiments.

[0095] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present invention with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. The elements in the claims will be interpreted broadly based on the language adopted in the claims, and are not limited to the examples described in this specification or during the implementation of this application, and the examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.

[0096] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, those of ordinary skill in the art can use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the present invention. This should not be interpreted as a feature of an invention that is not claimed for protection being necessary for any claim. On the contrary, the subject matter of the present invention may be less than all the features of the embodiments of a specific invention. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently used as a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the full scope of the equivalent forms of the attached claims and these claims.

Claims

1. A pointer instrument recognition method based on HRNet, characterized in that: The method comprises: Based on the instrument data set, the instrument recognition model is obtained through HRNet network training; The real-time image of the instrument is acquired, and the real-time image of the instrument is cropped according to a preset region of interest and then input into the instrument recognition model to obtain the instrument value.

2. The pointer instrument recognition method based on HRNet according to claim 1 is characterized in that: The instrument data set includes an instrument image and a corresponding pointer scale label; the pointer scale label includes a pointer base point coordinate, a pointer tail point coordinate, and a start scale coordinate to an end scale coordinate.

3. The pointer instrument recognition method based on HRNet according to claim 2 is characterized in that: Based on the instrument data set, the instrument recognition model is obtained through HRNet network training, including: The instrument images and corresponding pointer scale labels in the instrument dataset are imported into the HRNet network for training to obtain the instrument recognition model.

4. The pointer instrument recognition method based on HRNet according to claim 1 is characterized in that: The HRNet network includes multiple convolutional layers; wherein the output results of at least two convolutional layers are downsampled using a maximum pooling layer to reduce the size of the feature map by half.

5. The pointer instrument recognition method based on HRNet according to claim 4 is characterized in that: The HRNet network includes 20 convolutional layers; among them: The first and second layers are convolutional layers with a kernel of 3×3, a stride of 2, and 64 output channels; The third layer is a maximum pooling layer with a kernel of 2×2 and a stride of 2; From the 4th to the 7th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 64. Each convolutional layer is followed by normalization and activation; From the 8th to the 11th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 128; From the 12th to the 15th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 256; From the 16th to the 19th layer, each layer has 2 convolutional layers with a kernel of 3×3, a stride of 2, and an output channel of 512; The results of the 11th, 15th, and 19th stages are downsampled using a 2×2 max pooling layer to halve the feature map size; the output of each stage is fused with the output of the previous stage; The 20th layer is a fully connected layer with a kernel of 1×1 and an output channel number of 3×c, where c represents the number of key points.

6. The pointer instrument recognition method based on HRNet according to claim 1 is characterized in that: Acquire a real-time image of the instrument, crop the real-time image of the instrument according to a preset region of interest, and input the image into the instrument recognition model to obtain the instrument value, including: Collect real-time images, crop the images according to the preset regions of interest, and feed them into the instrument recognition model; The output of the instrument recognition model is obtained to obtain n vectors; each vector represents the detection result of a dashboard; each vector is composed of [x -1 ,y -1 ,x0,y0,x1,y1,x2,y2,…,x m ,y m ], which represent the horizontal and vertical coordinates of the pointer base point, the horizontal and vertical coordinates of the pointer tail point, and the horizontal and vertical coordinates of the first large scale to the mth large scale; extract the pointer base point R and tail point T, whose coordinates are [x -1 ,y -1 ], [x0, y0]; extract the scale coordinate P i , the coordinates are [x i ,y i ], where i = 1, 2, ... m, m is the number of scales; Calculate the scale index k that satisfies arg(RP k )<arg(RT)<arg(RP k+1 );RPk、RT、RPk+1 are the vectors from the pointer base point to the kth scale, the pointer tail point, and the k+1th scale respectively, arg represents the angle between the vector and the vertical axis; the proportional value a is calculated according to the angle between the vector corresponding to the scale index k and the vertical axis; the instrument value is calculated according to the proportional value a and the preset range L.

7. The pointer instrument recognition method based on HRNet according to claim 6 is characterized in that: According to the angle between the vector corresponding to the scale index k and the vertical axis, the scale value a is calculated by the following formula:

8. The pointer instrument recognition method based on HRNet according to claim 6 is characterized in that: According to the ratio value a and the preset range L, the instrument value is calculated by the following formula:

9. A pointer instrument recognition device based on HRNet, characterized in that: The device comprises: The model training unit is configured to obtain an instrument recognition model through HRNet network training based on the instrument data set; The value recognition unit is configured to obtain a real-time image of the instrument, crop the real-time image of the instrument according to a preset region of interest, and then input the cropped image into the instrument recognition model to obtain the instrument value. 10 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, perform the method according to claim 1 .

Images