Instrument detection and recognition method based on attention fusion and related device

Abstract

The present invention discloses an instrument detection and recognition method based on attention fusion and a related device. The method comprises the following steps: obtaining an environmental video; inputting a video frame in the environmental video into a YOLOV3‑tiny model based on attention fusion, and outputting a detection result, wherein the detection result includes an instrument classification result and coordinate information; obtaining an instrument image based on the detection result and preprocessing the instrument image to obtain a preprocessed instrument image; and recognizing the indication of a pointer-type instrument in the preprocessed instrument image. The present invention can accurately identify different instrument panels in a power distribution station and has a high detection rate.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an instrument detection and recognition method based on attention fusion and a related device. Background Art

[0002] At present, the urban distribution network has become a complex aggregate that carries massive operating information and complex power supply equipment. In addition, the construction of the distribution network covers a wide area and a long time span. The complexity of distribution network management has led to the fact that the management methods and efficiency of distribution equipment in some parts of my country cannot meet the power supply companies' demand for high reliability and high-quality services. The distribution station is responsible for the distribution of electric energy in the region, and the stable operation of the basic equipment and components of the power system is the premise of stable power supply in the distribution station. Usually, there are many instruments and meters such as opening and closing indicators, circuit breakers, discharge counters, and barometers in the distribution station, and they are distributed in every corner of the substation. The inspection personnel need to inspect the equipment of the entire substation in a high-voltage dangerous environment. It can be seen that the labor intensity is high, and it is difficult to ensure the high-quality completion of the inspection task in long-term high-intensity repetitive labor. Under the background of inefficient manual inspection work and harsh substation environment, there is an urgent need for an information recognition method for pointer instruments in the substation site to solve the problem of instrument reading recognition in the substation site. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes an instrument detection and identification method and related devices based on attention fusion. In the method and related devices, the automatic identification of instruments by the camera carried by the inspection robot can effectively replace manual labor and reduce labor. The inspection robot carries a camera to collect images of the instruments in the substation site, and then digital image processing technology can be used to detect and identify the instruments, and finally the results are output to the control system, thereby realizing unmanned monitoring of the instruments. Using this method to perform inspection tasks can greatly reduce the impact of human negligence or bad weather on the inspection quality, and effectively avoid accidents that may be caused by working in dangerous environments.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows:

[0005] The instrument detection and recognition method based on attention fusion includes the following steps:

[0006] Get the environment video;

[0007] Input the video frames in the environment video into the YOLOV3-tiny model based on attention fusion, and output the detection results, which include instrument classification results and coordinate information;

[0008] Acquire an instrument image based on the detection result and preprocess the instrument image to acquire a preprocessed instrument image;

[0009] Identify the readings of pointer instruments in preprocessed instrument images.

[0010] Preferably, the step of inputting the video frames in the environment video into the YOLOV3-tiny model based on attention fusion and outputting the detection results comprises the following steps:

[0011] The input video frame passes through a three-layer densely connected module and then undergoes convolution and maximum pooling processing, outputting different scale features F1, F2, and F3 in turn;

[0012] The output features F1, F2, and F3 of different scales are used as the input of the attention fusion module. The features F2 and F3 are upsampled respectively, and the upsampled features are channel-concatenated with the feature F1 to obtain the fused features. The fused features are processed by average pooling, 1*1 convolution, and sigmoid activation function, and then added to the same position elements of the feature F1 to obtain a feature map with attention weights.

[0013] Based on the feature map with attention weights, the instrument classification results and coordinate information are obtained.

[0014] Preferably, the preprocessing includes denoising and gamma correction.

[0015] Preferably, the gamma correction specifically comprises the following steps:

[0016] Multi-scale Gaussian function is used to extract illumination components;

[0017] Based on the extracted illumination components, adjusting parameters of the two-dimensional gamma function;

[0018] Brightness correction is performed on the instrument image based on the parameters of the two-dimensional gamma function.

[0019] Preferably, the identification of the indication of the pointer instrument in the instrument image after the processing specifically includes the following steps:

[0020] The processed instrument image is input into the convolutional neural network to extract the coordinates of key points in the instrument image, which include four points on the circumference of the instrument, the first end point of the pointer, the end point of the pointer and the center point of the instrument;

[0021] Based on the coordinates of key points in the instrument image, the readings of the pointer instrument are obtained.

[0022] Preferably, the convolutional neural network includes seven layers, the first layer interpolates and normalizes the instrument image to obtain an input image of 224×224×3; the second and third layers perform 3*3 convolution and pooling processing to obtain output images of 64×64×32 and 32×32×64, respectively; the fourth and fifth layers both perform 3*3 convolution processing to obtain output images of 16×16×64 and 2×2×128, respectively; the last two layers are fully connected layers; and the activation function in each layer adopts ReLU.

[0023] Preferably, the loss function of the convolutional neural network is:

[0024]

[0025] Where xi, yi represent the model output coordinates of the i-th key point, Represents the label value of the i-th key point, and N is the total number of samples.

[0026] Based on the above content, the present invention also discloses an instrument detection and recognition device based on attention fusion, including: a collection unit, a target determination unit, a preprocessing unit and a recognition unit, wherein:

[0027] The acquisition unit is used to acquire the environment video;

[0028] The target determination unit is used to input the video frame in the environment video into the YOLOV3-tiny model based on attention fusion, and output the detection result, wherein the detection result includes the instrument classification result and coordinate information;

[0029] The preprocessing unit is used to obtain an instrument image based on the detection result and preprocess the instrument image to obtain a preprocessed instrument image;

[0030] The recognition unit is used to recognize the indication of the pointer instrument in the preprocessed instrument image.

[0031] Based on the above content, the present invention further discloses a computer device, including: a memory for storing a computer program; and a processor for implementing any of the above methods when executing the computer program.

[0032] Based on the above content, the present invention further discloses a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any of the methods described above is implemented.

[0033] Based on the above technical solution, the beneficial effects of the present invention are:

[0034] 1) The present invention takes into account the uneven illumination and different shooting angles in the actual detection scene, which leads to unstable image quality. In order to reduce the impact of the external environment and random interference of the distribution station on the image, the present invention performs denoising on the image and pre-processes the image using a two-bit gamma function correction algorithm, thereby improving the accuracy of the instrument reading;

[0035] 2) The present invention aims at the balance between instrument panel detection rate and accuracy, and is successfully deployed in an embedded environment with limited computing resources. The present invention designs a lightweight instrument detection method, that is, based on YOLOV3-Tiny, a feature extraction network based on an improved YOLOV3-tiny based on the idea of ​​attention fusion features and dense connections is proposed. This method effectively fuses feature maps of different scales through the attention fusion module, effectively enhances the interaction of semantic information between feature maps, and improves the recognition accuracy of instruments;

[0036] 3) In order to solve the problem that there are many types of pointer instruments and the scale areas of the instruments vary greatly during the identification process of pointer instruments, making it impossible to use a unified method for identification, the present invention proposes a pointer meter real number recognition algorithm based on convolutional neural network key point detection, which can be applied to the algorithm for identifying various types of pointer instruments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is an application environment diagram of an instrument detection and recognition method based on attention fusion in an embodiment;

[0038] Figure 2 is a flow chart of an instrument detection and recognition method based on attention fusion in an embodiment;

[0039] Figure 3 is a schematic diagram of the structure of a YOLOV3-tiny model based on attention fusion in an instrument detection and recognition method based on attention fusion in an embodiment;

[0040] Figure 4 is a schematic diagram of the structure of a densely connected module in an instrument detection and recognition method based on attention fusion in an embodiment;

[0041] Figure 5 is a structural schematic diagram of an attention fusion module in an instrument detection and recognition method based on attention fusion in an embodiment;

[0042] Figure 6 is a schematic diagram of median filtering denoising in an instrument detection and recognition method based on attention fusion in an embodiment;

[0043] Figure 7is a schematic diagram of the structure of a convolutional neural network in an instrument detection and recognition method based on attention fusion in an embodiment;

[0044] Figure 8 is a schematic diagram of the structure of an instrument detection and recognition device based on attention fusion in an embodiment;

[0045] Fig. 9 The figure is a diagram of the internal structure of a computer device in an embodiment. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0047] The instrument detection and recognition method based on attention fusion provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment includes a computer device 110. The computer device 110 can obtain an environment video; the computer device 110 can input a video frame in the environment video into a YOLOV3-tiny model based on attention fusion, and output a detection result, wherein the detection result includes an instrument classification result and coordinate information; the computer device 110 can obtain an instrument image based on the detection result and preprocess the instrument image to obtain a preprocessed instrument image; the computer device 110 can identify the indication of a pointer instrument in the preprocessed instrument image. The computer device 110 can be, but is not limited to, various personal computers, laptops, inspection robots, tablet computers, and other devices.

[0048] In one embodiment, Figure 2 As shown, an instrument detection and recognition method based on attention fusion is provided, comprising the following steps:

[0049] Step 202, obtaining an environment video.

[0050] The computer device can be a patrol robot, which collects the distribution station environment through the camera on the patrol robot, finds the patrol point to obtain the environment video, and extracts the video frame as the input of the YOLOV3-tiny model based on attention fusion.

[0051] Step 204: input the video frames in the environment video into the YOLOV3-tiny model based on attention fusion, and output the detection results, which include instrument classification results and coordinate information.

[0052] The computer device passes the image into the backbone network of the YOLOV3-tiny model based on attention fusion. The input size of the image is 416×416. The structure of the YOLOV3-tiny model based on attention fusion is shown in Table 1.

[0053] Table 1 Structure of the YOLOV3-tiny model based on attention fusion

[0054]

[0055] See also Figure 3 , providing a structural diagram of the YOLOV3-tiny model based on attention fusion. Combined with the above table, the images collected by the camera are processed by convolution and maximum pooling after passing through 3 layers of densely connected modules, and the features F1 (data size is 104*104*32), F2 (data size is 52*52*64), and F3 (data size is 26*26*128) of different scales are output in turn; the output features F1, F2, and F3 of different scales are used as the input of the attention fusion module, and the features F2 and F3 are upsampled respectively, and the upsampled features are channel-concatenated with the feature F1 to obtain the fusion feature; the fusion feature is processed by average pooling, 1*1 convolution and sigmoid activation function, and then the same-position elements are added with the feature F1 to obtain the feature map with attention weights; based on the feature map with attention weights, the instrument classification results and coordinate information are obtained.

[0056] The design structure of densely connected modules is as follows Figure 4 As shown in the figure, a new feature extraction network is designed using three layers of densely connected modules, which does not bring complex calculation problems while deepening the convolution layer. From the beginning, the shallow features and deep features of the model are fully integrated. In the forward propagation of the network, the effective features of targets of different scales are not easily lost, which is conducive to the combination of deep convolution layer upsampling and shallow convolution layer in regression prediction, thereby enhancing the feature extraction ability of the entire network. The output of each dense module is the input of the attention fusion module.

[0057] The specific design of the attention fusion module is as follows Figure 5 As shown in the figure, in the design of the attention fusion module, the input is the output of three densely connected modules. Due to the different sizes of the feature maps, it is necessary to perform feature upsampling on input 2 (feature F2) and input 3 (feature F2) and then perform channel cascading to obtain the fusion feature. The specific operation can be expressed by the formula:

[0058] u=Concat{Conv(F1),UPConv(F2),UPConv(F3)}

[0059] Through channel cascading, the channel information fusion between feature maps of different scales is effectively realized, and average pooling is added after the attention fusion module. At this time, the feature map obtained contains the global receptive field. The role of Sigmoid is to map the input to [0, 1] to achieve weight normalization. In order to further increase the detailed information of shallow features, the same-position addition method is used at the end of the model to further realize the fusion of global contextual semantic information and improve the recognition accuracy of the model.

[0060] Step 206: acquiring an instrument image based on the detection result and preprocessing the instrument image to acquire a preprocessed instrument image.

[0061] Since most pointer instruments in the distribution station are made of glass, the collected instrument images are easily affected by light sources and angles, which leads to fluctuations in image quality and affects subsequent recognition effects. Therefore, it is necessary to preprocess the images appropriately. Image preprocessing can effectively provide more obvious features for image analysis and recognition. It mainly includes denoising and gamma correction, which are described as follows:

[0062] Denoising: The median filter method is used. The median filter has greatly improved the problem of image blur caused by the mean filter, but the too large or too small smoothing window will have a certain impact on the image quality. Therefore, the design of the present invention adopts the median filter denoising under the 5×5 square window, which will not cause too much loss of details while removing noise.

[0063] Median filtering is a typical nonlinear filtering method. Similar to the mean filtering method, it uses a template (smoothing) window, which is generally 3×3, 5×5, 7×7, etc. It sorts the pixel values ​​in the template from large to small, and replaces the central element value with the re-sorted median value to achieve the purpose of denoising. Figure 6 shown.

[0064] Gamma correction: Use the gamma function (curve) to correct the brightness of the image. The formula is as follows:

[0065]

[0066] Where: S(x,y) represents a given image. The whole process can be seen as three main steps: 1. Normalization: convert the pixel values ​​in the image to the interval [0, 1]; 2. Pre-compensation: obtain the normalized pixel values ​​to use the parameters is the compensation result of the exponent; 3. Denormalization: convert the pixel value back to an integer between [0, 255].

[0067] From the above formula, we can see that when γ>1, the dynamic range of the brighter part of the image is compressed, and the dynamic range of the darker part of the image is expanded, but the overall brightness of the image is reduced; when γ<1, the dynamic range of the brighter part is expanded, and the dynamic range of the darker part of the image is compressed, but the overall brightness of the image is improved. It can be seen that although gamma correction can use the γ value to adjust the brightness, when the illumination conditions of different images are inconsistent, it is necessary to constantly change the γ value manually. Therefore, we want to find a solution to let it adaptively adjust the γ value according to the distribution of the illumination components of the image, so as to correct the uneven illumination of the image. Therefore, for images with uneven illumination, by designing the illumination extraction component, according to the illumination distribution characteristics, the brightness distribution of the image is adaptively corrected.

[0068] A multi-scale Gaussian function is used to extract the illumination component, as shown in the following formula:

[0069]

[0070] Where M represents the number of Gaussian center surround functions, that is, the number of selected scales. Considering the balance between the effect of the extracted illumination component and the computational efficiency, a three-scale Gaussian function is selected, that is, M = 3; w i The weight coefficients corresponding to the various scales represented by are set as It is the estimated value of the illumination component extracted by the three-scale Gaussian function, S(x, y) represents a given image, and F(x, y) represents the convolution operation.

[0071] After the illumination component is extracted, the brightness distribution of the image can be adaptively corrected according to the distribution characteristics of the illumination. Gamma correction can realize the brightness correction of the image, appropriately increase the brightness of the weak illumination area and reduce the brightness of the strong illumination area. Therefore, the parameters of the two-dimensional gamma function are adjusted by using the illumination distribution characteristics, as shown below:

[0072]

[0073] In the formula, represents the corrected image. γ is the exponential value of brightness correction. m is the mean brightness of the illumination component. According to the principle of gamma correction, The value is set by the light intensity. The smaller the value, the brighter the image will be after correction. The larger the value, the lower the brightness of the corrected image.

[0074] Step 208, identifying the reading of the pointer instrument in the preprocessed instrument image.

[0075] In this embodiment, the processed instrument image is input into the convolutional neural network to extract the coordinates of key points in the instrument image, which include four points on the circumference of the instrument, the first end point of the pointer, the end point of the pointer and the center point of the instrument. The structure of the convolutional neural network is as follows: Figure 7 shown.

[0076] In order to reduce the amount of data and ensure the details of the input image, the convolutional neural network converts the processed instrument image into a 224×224 image through interpolation calculation, and divides it by 255 for normalization as input data, and outputs a one-dimensional vector consisting of 7 coordinates. The first three layers use one convolution and pooling for processing, the fourth and fifth layers only use one convolution, and the last two layers are fully connected layers, and the linear rectification function (ReLU) is uniformly used as the activation function of each layer.

[0077] The loss function of the convolutional neural network is:

[0078]

[0079] Where xi,yi represents the model output coordinates of the i-th key point, Represents the label value of the i-th key point, and N is the total number of samples.

[0080] By designing a pointer table real number recognition algorithm based on convolutional neural network key point detection, the test results of the entire model for key point detection are shown in Table 2.

[0081] Table 2 Test results of key point detection

[0082]

[0083] Based on the test results of key point detection, it can be seen that the accuracy of convolutional neural network for key point detection can reach more than 95%.

[0084] In order to further test the correctness of the pointer reading, the reading result calculated by the algorithm is compared with the marked value, and the meter reading is measured at different time points in the distribution station. If the difference does not exceed 5% of the maximum range, the algorithm reading is considered correct. The test results are shown in Table 3.

[0085] Table 3 Instrument reading test results at different time points

[0086]

[0087] It can be seen that this method can correctly identify the instrument panel indications with an accuracy of more than 91%.

[0088] In one embodiment, Figure 8As shown, an instrument detection and recognition device 300 based on attention fusion is provided, comprising: a collection unit 310, a target determination unit 320, a preprocessing unit 330 and a recognition unit 340, wherein:

[0089] The acquisition unit 310 is used to acquire the environment video;

[0090] The target determination unit 320 is used to input the video frame in the environment video into the YOLOV3-tiny model based on attention fusion, and output the detection result, wherein the detection result includes the instrument classification result and coordinate information;

[0091] The preprocessing unit 330 is used to obtain an instrument image based on the detection result and preprocess the instrument image to obtain a preprocessed instrument image;

[0092] The recognition unit 340 is used to recognize the indication of the pointer instrument in the preprocessed instrument image.

[0093] In one embodiment, the target determination unit 320 is also used to perform convolution and maximum pooling on the input video frame after passing it through a three-layer densely connected module, and output different-scale features F1, F2, and F3 in sequence; the output different-scale features F1, F2, and F3 are used as inputs of the attention fusion module, and features F2 and F3 are upsampled respectively, and the upsampled features are channel-concatenated with feature F1 to obtain fused features; the fused features are average pooled, 1*1 convolved, and processed with a sigmoid activation function, and then the same-position elements are added to feature F1 to obtain a feature map with attention weights; based on the feature map with attention weights, the instrument classification results and coordinate information are obtained.

[0094] In one embodiment, the preprocessing unit 330 is also used to perform denoising and gamma correction. The specific steps of gamma correction are: extracting illumination components using a multi-scale Gaussian function; adjusting the parameters of the two-dimensional gamma function based on the extracted illumination components; and performing brightness correction on the instrument image based on the parameters of the two-dimensional gamma function.

[0095] In one embodiment, the recognition unit 340 is also used to input the processed instrument image into a convolutional neural network to extract the coordinates of key points in the instrument image, wherein the key point coordinates include four points on the circumference of the instrument, the first end point of the pointer, the end point of the pointer and the center point of the instrument; based on the key point coordinates in the instrument image, the reading of the pointer instrument is obtained.

[0096] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an instrument detection and recognition method based on attention fusion is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0097] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0099] Get the environment video;

[0100] Input the video frames in the environment video into the YOLOV3-tiny model based on attention fusion, and output the detection results, which include instrument classification results and coordinate information;

[0101] Acquire an instrument image based on the detection result and preprocess the instrument image to acquire a preprocessed instrument image;

[0102] Identify the readings of pointer instruments in preprocessed instrument images.

[0103] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the input video frame is convolved and max-pooled after passing through a 3-layer densely connected module, and features F1, F2, and F3 of different scales are output in sequence; the output features F1, F2, and F3 of different scales are used as inputs of an attention fusion module, features F2 and F3 are up-sampled respectively, and the up-sampled features are channel-concatenated with feature F1 to obtain fused features; the fused features are average-pooled, 1*1 convolved, and processed with a sigmoid activation function, and then the same-position elements are added to feature F1 to obtain a feature map with attention weights; based on the feature map with attention weights, instrument classification results and coordinate information are obtained.

[0104] In one embodiment, when the processor executes the computer program, the following steps are also implemented: denoising and gamma correction. The specific steps of gamma correction are: extracting illumination components using a multi-scale Gaussian function; adjusting the parameters of the two-dimensional gamma function based on the extracted illumination components; and performing brightness correction on the instrument image based on the parameters of the two-dimensional gamma function.

[0105] In one embodiment, when the processor executes the computer program, the following steps are also implemented: inputting the processed instrument image into a convolutional neural network, extracting the coordinates of key points in the instrument image, wherein the key point coordinates include four points on the circumference of the instrument, the first end point of the pointer, the end point of the pointer and the center point of the instrument; based on the coordinates of the key points in the instrument image, obtaining the indication of the pointer instrument.

[0106] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0107] Get the environment video;

[0108] Input the video frames in the environment video into the YOLOV3-tiny model based on attention fusion, and output the detection results, which include instrument classification results and coordinate information;

[0109] Acquire an instrument image based on the detection result and preprocess the instrument image to acquire a preprocessed instrument image;

[0110] Identify the readings of pointer instruments in preprocessed instrument images.

[0111] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the input video frame is convolved and max-pooled after passing through a 3-layer densely connected module, and features F1, F2, and F3 of different scales are output in sequence; the output features F1, F2, and F3 of different scales are used as inputs of an attention fusion module, features F2 and F3 are up-sampled respectively, and the up-sampled features are channel-concatenated with feature F1 to obtain fused features; the fused features are average-pooled, 1*1 convolved, and processed with a sigmoid activation function, and then the same-position elements are added to feature F1 to obtain a feature map with attention weights; based on the feature map with attention weights, instrument classification results and coordinate information are obtained.

[0112] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: denoising and gamma correction. The specific steps of gamma correction are: extracting illumination components using a multi-scale Gaussian function; adjusting the parameters of the two-dimensional gamma function based on the extracted illumination components; and performing brightness correction on the instrument image based on the parameters of the two-dimensional gamma function.

[0113] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the processed instrument image into a convolutional neural network, extracting the coordinates of key points in the instrument image, wherein the key point coordinates include four points on the circumference of the instrument, the first end point of the pointer, the end point of the pointer and the center point of the instrument; based on the key point coordinates in the instrument image, obtaining the reading of the pointer instrument.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. The instrument detection and recognition method based on attention fusion is characterized by: The following steps are involved: Get the environment video; The video frames in the environment video are input into the YOLOV3-tiny model based on attention fusion, and the detection results are output, which specifically includes the following steps: the input video frame is processed by convolution and maximum pooling after passing through a 3-layer dense connection module, and different scale features F1, F2, and F3 are output in sequence; the output different scale features F1, F2, and F3 are used as the input of the attention fusion module, and the features F2 and F3 are upsampled respectively, and the upsampled features are channel-concatenated with the feature F1 to obtain the fusion feature; the fusion feature is processed by average pooling, 1*1 convolution and sigmoid activation function, and then the same position elements are added with the feature F1 to obtain a feature map with attention weights; Based on the feature map with attention weights, obtain the instrument classification results and coordinate information; Acquire an instrument image based on the detection result and preprocess the instrument image to acquire a preprocessed instrument image; Identify the readings of pointer instruments in preprocessed instrument images.

2. The instrument detection and recognition method based on attention fusion according to claim 1 is characterized in that: The preprocessing includes denoising and gamma correction.

3. The instrument detection and recognition method based on attention fusion according to claim 2 is characterized in that: The gamma correction specifically includes the following steps: Multi-scale Gaussian function is used to extract illumination components; Based on the extracted illumination components, adjusting parameters of the two-dimensional gamma function; Brightness correction is performed on the instrument image based on the parameters of the two-dimensional gamma function.

4. The instrument detection and recognition method based on attention fusion according to claim 1 is characterized in that: The method of identifying the indication of the pointer instrument in the preprocessed instrument image specifically comprises the following steps: The processed instrument image is input into the convolutional neural network to extract the coordinates of key points in the instrument image, which include four points on the circumference of the instrument, the first end point of the pointer, the end point of the pointer and the center point of the instrument; Based on the coordinates of key points in the instrument image, the readings of the pointer instrument are obtained.

5. The instrument detection and recognition method based on attention fusion according to claim 4 is characterized in that: The convolutional neural network includes seven layers, and the first layer performs interpolation and normalization processing on the instrument image to obtain an input image of 224×224×3; The second and third layers perform 3*3 convolution and pooling processing to obtain output images of 64×64×32 and 32×32×64 respectively; the fourth and fifth layers perform 3*3 convolution processing to obtain output images of 16×16×64 and 2×2×128 respectively; the last two layers are fully connected layers; the activation function in each layer is ReLU.

6. The instrument detection and recognition method based on attention fusion according to claim 4 is characterized in that: The loss function of the convolutional neural network is: Where xi, yi represent the model output coordinates of the i-th key point, Represents the label value of the i-th key point, and N is the total number of samples.

7. The instrument detection and recognition device based on attention fusion is characterized in that: include: A collection unit, a target determination unit, a preprocessing unit and a recognition unit, wherein: The acquisition unit is used to acquire the environment video; The target determination unit is used to input the video frame in the environment video into the YOLOV3-tiny model based on attention fusion, and output the detection result, which specifically includes the following steps: the input video frame is subjected to convolution and maximum pooling processing after passing through the three-layer dense connection module, and the different scale features F1, F2, and F3 are output in sequence; the output different scale features F1, F2, and F3 are used as the input of the attention fusion module, and the features F2 and F3 are upsampled respectively, and the upsampled features are channel-concatenated with the feature F1 to obtain the fusion feature; the fusion feature is average-pooled, 1*1 convolution and sigmoid activation function are processed, and the same-position elements are added to the feature F1 to obtain a feature map with attention weights; based on the feature map with attention weights, the instrument classification result and coordinate information are obtained; The preprocessing unit is used to obtain an instrument image based on the detection result and preprocess the instrument image to obtain a preprocessed instrument image; The recognition unit is used to recognize the indication of the pointer instrument in the preprocessed instrument image.

8. A computer device, characterized in that: The device comprises: a memory for storing a computer program; A processor, configured to implement the method according to any one of claims 1 to 6 when executing the computer program.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images