A dashboard intelligent detection and reading method based on deep learning

Abstract

This invention discloses a deep learning-based intelligent detection and reading method for dashboards, belonging to the field of dashboard detection technology. It is highly intelligent and automated, achieving simultaneous recognition of the dashboard body and scale markings through multi-target collaborative detection technology. No manual configuration of parameters such as scale range is required, and the system's automation level reaches an industry-leading level. Compared to traditional methods, deployment efficiency is improved by 80%, and maintenance costs are reduced by 60%. It boasts excellent recognition accuracy, employing an integrated OCR recognition engine and intelligent voting mechanism, achieving a scale value recognition accuracy of over 98.5% even in challenging scenarios such as complex lighting, small fonts, and low resolution. The geometrically constrained pointer positioning algorithm enables angle measurement accuracy to reach ±0.5°, and reading errors are controlled within ±1% of full scale, meeting the requirements of high-precision industrial applications.

Description

Technical Field

[0001] This invention relates to the field of dashboard detection technology, and in particular to a deep learning-based intelligent dashboard detection and reading method. Background Technology

[0002] Instrument reading recognition is a crucial technological requirement in fields such as autonomous robot inspection and industrial robot sensing. In applications like power line inspection robots, pipeline inspection robots, and factory automation robots, robots need to recognize the readings of numerous analog pointer instruments in real time. However, due to changes in perspective caused by robot movement, end-effector jitter, and interference from complex industrial environments, the automatic acquisition of instrument readings has long faced challenges such as low accuracy, poor real-time performance, and weak generalization ability.

[0003] Traditional meter reading methods are insufficient to meet the practical needs of robotic systems. Manual meter reading defeats the purpose of autonomous robot operation; simple threshold segmentation methods based on image processing are highly sensitive to changes in lighting and motion blur during robot movement, lacking robustness; template matching methods based on traditional machine vision require pre-setting templates for different meter types, failing to adapt to the flexible deployment requirements of robots in multiple scenarios and tasks. With the development of deep learning technology, target detection methods based on convolutional neural networks provide a new technical path for robot meter recognition. However, existing methods still have significant shortcomings in multi-target collaborative detection under dynamic robot perspectives, adaptive capabilities in complex industrial scenarios, and precise localization of small targets, requiring further research and breakthroughs.

[0004] The prior art provides "an automatic reading method for pointer instruments based on image recognition CN108428244A". This method uses Hough circle detection to locate the dial and Hough line detection to identify the pointer. However, this method has high requirements for image quality. The recognition rate drops significantly under conditions such as uneven lighting and blurred pointers, and it cannot simultaneously handle the automatic recognition of scale values.

[0005] The existing technology provides a "Dashboard Automatic Recognition System CN110443143A Based on Deep Learning". This system uses a single target detection network to recognize the entire dashboard area and then extracts the pointer angle through post-processing. However, it fails to achieve multi-target joint detection of the dashboard, maximum value and minimum value, which means that the scale range needs to be manually configured and the degree of automation is insufficient.

[0006] The existing technology provides "a smart recognition method for pointer instruments CN109978848A". This method uses traditional image processing algorithms for pointer detection and adopts a fixed angle calculation formula. However, the accuracy is low in complex scenarios such as dial perspective distortion, pointer occlusion, and scale wear. In addition, the OCR recognition module is single and has poor adaptability to scale values ​​of different fonts and sizes.

[0007] In summary, analysis of current patents and technical solutions reveals the following drawbacks: (1) Existing methods mostly adopt single-target detection strategies, which cannot achieve coordinated and accurate positioning of multiple key elements such as the instrument panel body and scale markings, resulting in insufficient system automation and requiring manual intervention to configure parameters such as scale range; (2) Traditional OCR recognition methods have low accuracy in recognizing instrument scale values, especially under complex conditions such as small fonts, low resolution, and uneven lighting, which are prone to misrecognition. They also lack multi-engine collaboration and voting mechanisms. (3) The pointer detection algorithm is not robust enough to occlusion, blurring, low contrast and other conditions, and the angle calculation method does not fully consider the geometric characteristics and perspective distortion of the dial, resulting in a large reading error; (4) The existing system lacks a complete end-to-end solution, and the integration between functional modules is low, making it difficult to form a unified technical framework and standardized processing flow. Summary of the Invention

[0008] In view of the above problems, this invention provides a deep learning-based intelligent detection and reading method for dashboards to overcome or at least partially solve the above problems. By employing advanced multi-label target detection technology, an integrated OCR recognition engine, and a geometrically constrained pointer positioning algorithm, it achieves fully automatic, high-precision reading of dashboards, solving the problems of low recognition rate and insufficient automation in traditional methods under complex industrial scenarios.

[0009] This invention provides the following solution: A deep learning-based intelligent detection and reading method for dashboards includes: A multi-target collaborative detection model is constructed, which simultaneously detects the dashboard main body, the maximum scale value label, and the minimum scale value label through a single deep learning forward propagation, and obtains the bounding box coordinates of the three types of targets; An integrated OCR recognition engine is used to generate multiple image preprocessing variants for the detected maximum and minimum scale value regions, and multiple OCR engines are called in parallel for recognition. The final scale value is selected through a weighted voting mechanism. Pointer positioning with geometric constraints is performed by constructing an annular region of interest centered on the detected dial center. Pointer line segments are detected within the annular region of interest, and the pointer line segments are optimized by a scoring function, which is the line segment length divided by the distance from the pointer line segment to the center of the circle. Angle calculation and reading conversion are performed based on the geometric relationship of the scale. The absolute angle of the scale mark is calculated according to the center coordinates of the detection frame based on the minimum and maximum scale values. After calculating the absolute angle of the pointer tip and processing the angle difference across zero points, the instrument reading is obtained by linear interpolation formula based on the angle ratio relationship.

[0010] Preferably, the multi-target cooperative detection model includes: DarkNet-53 backbone feature extraction network, and introduces C2f module to replace C3 module; The path aggregation feature pyramid network employs a bidirectional feature fusion mechanism that combines top-down and bottom-up approaches. The decoupled multi-task detection head adopts an anchorless prediction paradigm and eliminates the objectness branch.

[0011] Preferably, the training process of the multi-target cooperative detection model includes: A header region annotation strategy is adopted to select and annotate the header region of the scale numbers; A phased fine-tuning training strategy was adopted, freezing the backbone network parameters for the first 20 training cycles, unfreezing the entire network for the next 50 training cycles, and disabling Mosaic data augmentation for the last 10 training cycles. The overall loss function for training is expressed by the following formula:

[0012] In the formula: Represents classification loss, Indicates regression loss, This represents the balancing hyperparameter.

[0013] Preferably, the plurality of OCR engines include at least EasyOCR, Tesseract, and PaddleOCR; The various image preprocessing variants include at least CLAHE contrast enhancement, adaptive threshold binarization, Otsu global threshold segmentation, color inversion, morphological closing operation, and sharpening filtering.

[0014] Preferably, in the weighted voting mechanism, the formula for calculating the comprehensive score of each candidate number is:

[0015] The candidate number with the highest score is selected as the final recognition result, and the recognition result is validated for digital validity. Only integers, decimals, or decimals with leading zeros are accepted.

[0016] Preferably, the construction parameters of the annular area of ​​interest include an inner diameter equal to 0.2 times the dial radius and an outer diameter equal to 0.95 times the dial radius; The detection of the pointer line segment adopts the probabilistic Hough linear transformation, and the minimum line segment threshold is set to be no less than 0.2 times the dial radius; The calculation formula for the scoring function is as follows:

[0017] Choose the line with the highest score as the pointer.

[0018] Preferably, the detection of the dial center employs a multi-strategy fusion method: Hough circle transform is preferred for testing standard circular dials; If the Hough circle transform detection fails, an edge detection combined with an ellipse fitting algorithm is used. If ellipse fitting fails, the center of the image will be used as the center of the dial, and 40% of the shorter side of the image will be used as the radius of the dial.

[0019] Preferably, the pointer positioning of the geometric constraint further includes a backup detection mechanism, which includes extracting the largest connected component within the annular region and calculating the direction angle of the smallest bounding rectangle of the connected component as the pointer angle when the probabilistic Hough line detection fails to detect a valid pointer line segment.

[0020] Preferably, the angle calculation and reading conversion specifically includes: Establish a standardized angular coordinate system with the center of the dial as the origin, the vertical downward direction as the 0° reference, the clockwise direction as the direction of angle increase, and the angle range as 0°~360°; Calculate the absolute angle of the scale markings. Based on the center coordinates of the detection boxes for the minimum and maximum scale values, calculate the angular position of the scale relative to the center of the dial to obtain the geometric distribution information of the scale. Calculate the absolute angle of the pointer. Based on the coordinates of the endpoints of the pointer line segment, identify the endpoint farther from the center of the circle as the pointer tip, and calculate the angle of the pointer tip relative to the center of the circle. The algorithm for calculating angle differences across zero points is designed to handle cases where the angle crosses the 0° / 360° boundary. The clockwise angle range from the minimum to the maximum scale is calculated, as well as the clockwise angle offset of the pointer relative to the minimum scale. Based on the angular ratio, a linear interpolation formula is used to calculate the instrument reading; the linear interpolation formula is as follows:

[0021] Apply boundary constraints to the calculation results to keep them between the minimum and maximum scale values.

[0022] Preferably, it also includes visualization and result output steps: Generate multi-level visualization results: draw detection boxes and label information on the original image, and mark the positions of the dashboard, maximum value, and minimum value; draw the center of the circle, the outline of the dashboard, the pointer line segment, and the scale reference line on the cropped dashboard image; and overlay and display the recognized values ​​and calculated angle information. Generate a structured result dictionary, which includes at least the read value, pointer angle, scale range, angle range, and center coordinates.

[0023] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This application provides a deep learning-based intelligent detection and reading method for dashboards. This highly intelligent and automated method utilizes multi-target collaborative detection technology to achieve simultaneous recognition of the dashboard body and scale markings, eliminating the need for manual configuration of parameters such as scale ranges. The system's automation level is industry-leading. Compared to traditional methods, deployment efficiency is improved by 80%, and maintenance costs are reduced by 60%.

[0024] It boasts exceptional recognition accuracy, employing an integrated OCR recognition engine and intelligent voting mechanism. Even in challenging scenarios such as complex lighting, small fonts, and low resolution, the scale value recognition accuracy reaches over 98.5%. The geometrically constrained pointer positioning algorithm enables angle measurement accuracy to reach ±0.5°, with reading errors controlled within ±1% of full scale, meeting the demands of high-precision industrial applications.

[0025] With strong environmental adaptability, the system can effectively cope with complex working conditions such as perspective distortion, uneven lighting, blurred pointers, and partial occlusion through innovative designs such as multi-strategy fusion dial detection, diversified image preprocessing, and backup detection mechanisms. The success rate of recognition in actual industrial environments reaches more than 95%, which is significantly better than the 70%-80% success rate of traditional methods.

[0026] It boasts excellent real-time performance, based on a lightweight deep learning model and optimized processing flow. The end-to-end processing time for a single image is less than 200 milliseconds. It supports real-time video stream analysis and can be deployed on edge computing devices to meet the real-time monitoring needs of industrial sites.

[0027] It has broad application prospects, good versatility and scalability. It is not only suitable for conventional instruments such as pressure gauges and thermometers, but can also be extended to various meter types such as electricity meters, water meters, and gas meters. It has broad application prospects in industries such as power, petroleum, chemical, manufacturing, and municipal engineering. It can effectively solve the pain points of low efficiency and easy error in traditional manual meter reading.

[0028] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0030] Figure 1 This is a flowchart of a deep learning-based intelligent detection and reading method for dashboards provided in an embodiment of the present invention; Figure 2 This is a flowchart of a complete instrument reading recognition process provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of a deep learning-based intelligent detection and reading device for dashboards provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a deep learning-based intelligent detection and reading device for dashboards provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0032] See Figure 1 This invention provides a method for intelligent detection and reading of dashboards based on deep learning, as an embodiment of the present invention. Figure 1 As shown, the method may include: S101: Construct a multi-target collaborative detection model that simultaneously detects the dashboard main body, the maximum scale value label, and the minimum scale value label through a single deep learning forward propagation, obtaining the bounding box coordinates of the three types of targets; in specific implementation, the multi-target collaborative detection model provided in this application embodiment may include: DarkNet-53 backbone feature extraction network, and introduces C2f module to replace C3 module; The path aggregation feature pyramid network employs a bidirectional feature fusion mechanism that combines top-down and bottom-up approaches. The decoupled multi-task detection head adopts an anchorless prediction paradigm and eliminates the objectness branch.

[0033] The training process of the multi-target collaborative detection model includes: A header region annotation strategy is adopted to select and annotate the header region of the scale numbers; A phased fine-tuning training strategy was adopted, freezing the backbone network parameters for the first 20 training cycles, unfreezing the entire network for the next 50 training cycles, and disabling Mosaic data augmentation for the last 10 training cycles. The overall loss function for training is expressed by the following formula:

[0034] In the formula: Represents classification loss, Indicates regression loss, This represents the balancing hyperparameter.

[0035] S102: An integrated OCR recognition engine is used to generate multiple image preprocessing variants for the detected maximum and minimum scale value regions, and multiple OCR engines are called in parallel for recognition. The final scale value is selected through a weighted voting mechanism. In specific implementation, the embodiments of this application may provide multiple OCR engines, including at least EasyOCR, Tesseract and PaddleOCR. The various image preprocessing variants include at least CLAHE contrast enhancement, adaptive threshold binarization, Otsu global threshold segmentation, color inversion, morphological closing operation, and sharpening filtering.

[0036] In the weighted voting mechanism, the formula for calculating the comprehensive score of each candidate number is as follows:

[0037] The candidate number with the highest score is selected as the final recognition result, and the recognition result is validated for digital validity. Only integers, decimals, or decimals with leading zeros are accepted.

[0038] S103: Perform pointer positioning with geometric constraints, construct an annular region of interest centered on the detected dial center, detect pointer line segments within the annular region of interest, and optimize pointer line segments using a scoring function, where the scoring function is the line segment length divided by the distance from the pointer line segment to the center of the circle; In specific implementation, embodiments of this application may provide the construction parameters of the annular region of interest including an inner diameter equal to 0.2 times the dial radius and an outer diameter equal to 0.95 times the dial radius; The detection of the pointer line segment adopts the probabilistic Hough linear transformation, and the minimum line segment threshold is set to be no less than 0.2 times the dial radius; The calculation formula for the scoring function is as follows:

[0039] Choose the line with the highest score as the pointer.

[0040] The detection of the center of the dial uses a multi-strategy fusion method: Hough circle transform is preferred for testing standard circular dials; If the Hough circle transform detection fails, an edge detection combined with an ellipse fitting algorithm is used. If ellipse fitting fails, the center of the image will be used as the center of the dial, and 40% of the shorter side of the image will be used as the radius of the dial.

[0041] The pointer positioning of the geometric constraints also includes a backup detection mechanism, which includes extracting the largest connected component within the annular region and calculating the direction angle of the smallest bounding rectangle of the connected component as the pointer angle when the probabilistic Hough line detection fails to detect a valid pointer line segment.

[0042] S104: Angle calculation and reading conversion based on scale geometry. The absolute angle of the scale mark is calculated based on the center coordinates of the detection frame according to the minimum and maximum scale values. After calculating the absolute angle of the pointer tip and processing the angle difference across zero points, the instrument reading is obtained by linear interpolation formula based on the angle ratio relationship.

[0043] In specific implementation, the angle calculation and reading conversion provided in this application embodiment specifically includes: Establish a standardized angular coordinate system with the center of the dial as the origin, the vertical downward direction as the 0° reference, the clockwise direction as the direction of angle increase, and the angle range as 0°~360°; Calculate the absolute angle of the scale markings. Based on the center coordinates of the detection boxes for the minimum and maximum scale values, calculate the angular position of the scale relative to the center of the dial to obtain the geometric distribution information of the scale. Calculate the absolute angle of the pointer. Based on the coordinates of the endpoints of the pointer line segment, identify the endpoint farther from the center of the circle as the pointer tip, and calculate the angle of the pointer tip relative to the center of the circle. The algorithm for calculating angle differences across zero points is designed to handle cases where the angle crosses the 0° / 360° boundary. The clockwise angle range from the minimum to the maximum scale is calculated, as well as the clockwise angle offset of the pointer relative to the minimum scale. Based on the angular ratio, a linear interpolation formula is used to calculate the instrument reading; the linear interpolation formula is as follows:

[0044] Apply boundary constraints to the calculation results to keep them between the minimum and maximum scale values.

[0045] S105: Visualization and Results Output: Generate multi-level visualization results: draw detection boxes and label information on the original image, and mark the positions of the dashboard, maximum value, and minimum value; draw the center of the circle, the outline of the dashboard, the pointer line segment, and the scale reference line on the cropped dashboard image; and overlay and display the recognized values ​​and calculated angle information. Generate a structured result dictionary, which includes at least the read value, pointer angle, scale range, angle range, and center coordinates.

[0046] This invention discloses a method for multi-target collaborative recognition and accurate reading of smart dashboards based on deep learning. It aims to achieve fully automatic high-precision reading of dashboards by using advanced multi-label target detection technology, integrated OCR recognition engine and geometrically constrained pointer positioning algorithm, thereby solving the problems of low recognition rate and insufficient automation of traditional methods in complex industrial scenarios.

[0047] This invention proposes a method for multi-target collaborative recognition and accurate reading of intelligent dashboards based on deep learning. It forms a complete end-to-end technical solution, from the construction of multi-target detection model, the design of integrated OCR recognition engine, the pointer positioning algorithm with geometric constraints, to angle calculation and reading conversion.

[0048] By constructing a multi-target collaborative detection model, the main body of the dashboard, the maximum scale value, and the minimum scale value can be accurately located simultaneously in a single inference, realizing the collaborative perception of key elements and improving the system's automation level. An integrated OCR recognition engine is designed, which integrates multiple OCR technologies and diverse preprocessing strategies. The optimal result is selected through an intelligent voting mechanism, significantly improving the accuracy and robustness of scale value recognition. A geometrically constrained pointer positioning algorithm is proposed, which effectively handles complex situations such as pointer ambiguity and occlusion through ring area constraints, multi-strategy fusion detection, and comprehensive scoring optimization. An angle calculation framework based on geometric relationships is established, which achieves accurate mapping from pointer angle to instrument reading through standardized coordinate system, cross-zero point angle processing, and linear interpolation transformation, avoiding the cumulative error of traditional trajectory tracking methods.

[0049] The method provided in this application adopts a multi-target collaborative detection architecture, which innovatively incorporates multiple heterogeneous targets such as the dashboard body and scale labels into a unified deep learning detection framework. Through an improved feature pyramid network and a decoupled detection head design, it achieves end-to-end collaborative perception and accurate localization, eliminates the information loss in multi-stage processing in traditional methods, and significantly improves the automation level and overall performance of the system.

[0050] An integrated multi-engine OCR recognition system was constructed to overcome the limitations of a single recognition engine. It generates rich image variations through diverse preprocessing strategies, integrates the parallel inference results of multiple OCR engines, and designs a weighted voting mechanism based on frequency and confidence. In challenging scenarios such as small fonts, low resolution, and complex lighting, the recognition accuracy is improved by more than 30% compared to a single method.

[0051] A geometrically constrained pointer localization algorithm is proposed. Through innovative techniques such as circular region of interest constraints, multi-strategy fusion detection, and comprehensive scoring optimization, it effectively solves problems such as pointer blurring, partial occlusion, and low contrast. Compared to the traditional Hough line detection method, this algorithm improves the detection success rate by 40% in complex scenes and increases the angle measurement accuracy to within ±0.5°.

[0052] A theoretical framework for angle calculation and reading conversion based on geometric invariance is established. The instrument reading is directly calculated through the spatial geometric relationship of the scale markings. This eliminates the dependence of traditional methods on pointer movement trajectory, fundamentally eliminating the cumulative error of trajectory tracking and ID switching problems. It is applicable to complex situations such as perspective distortion and non-standard ranges, and the reading error is controlled within ±1% of the full scale.

[0053] The following is a detailed description of the deep learning-based intelligent detection and reading method for dashboards provided in the embodiments of this application.

[0054] To address the shortcomings of existing technologies, this invention proposes a deep learning-based method for multi-target collaborative recognition and accurate reading of intelligent dashboards. The following is a detailed technical solution including the training process: S1: Constructing a multi-target collaborative detection model: S11: Design an improved deep convolutional neural network architecture. An enhanced DarkNet-53 is adopted as the backbone feature extraction network. By introducing a C2f module to replace the traditional C3 module, the gradient flow information is fully transmitted, and the feature representation capability is significantly improved. This backbone network, while maintaining reasonable computational complexity, possesses deeper network layers and stronger multi-scale feature extraction capabilities.

[0055] S12: Construct a path aggregation feature pyramid network. A bidirectional feature fusion mechanism is adopted at the network neck. Through feature propagation along two paths, top-down and bottom-up, the fine edge information of the lower level is deeply fused with the abstract semantic information of the higher level, which significantly enhances the network's ability to perceive targets of different scales, especially improving the detection accuracy of small target scale values.

[0056] S13: Design a decoupled multi-task detection head. A task-separated detection head architecture is adopted, eliminating the traditional objectness branch and reducing semantic conflicts between different detection tasks. An anchor-free prediction paradigm is used to simplify the model structure, reduce computational complexity, and improve inference speed.

[0057] S14: Implement a multi-label collaborative detection strategy. Establish a unified detection framework for three types of targets: the dashboard main body, the maximum scale value, and the minimum scale value. Acquire the location information of all key elements simultaneously through a single forward propagation to achieve end-to-end collaborative detection.

[0058] S2: Dataset Construction and Model Training S21: Construct a multi-source heterogeneous dataset. Image samples are extracted from industrial site monitoring videos at fixed frame intervals and combined with open-source datasets from the Internet to form a comprehensive training dataset covering various instrument types, different lighting conditions, and diverse shooting angles.

[0059] S22: A header region annotation strategy is adopted. Considering the characteristics of instrument scale annotations, the header region is used instead of the entire text region for annotation, reducing overlapping and occlusion of the annotation areas and improving the discrimination ability of the detection model. Professional annotation tools are used to accurately select the scale numbers in the image, generating standardized annotation files conforming to the YOLO format.

[0060] S23: Implement diverse data augmentation. Perform preprocessing operations such as horizontal flipping, mirror transformation, and scale normalization on the training images to uniformly scale the images to 640×640 resolution and perform pixel-level normalization to enhance the model's generalization ability.

[0061] S24: A phased fine-tuning training strategy is adopted. Transfer learning is performed based on a large-scale pre-trained model, and a differentiated learning rate scheduling scheme is designed: the learning rate of the backbone network is set to one-tenth of the base learning rate, while the learning rate of the detector head remains unchanged; the backbone network parameters are frozen for training in the first 20 epochs, and the entire network is unfrozen for training in the next 50 epochs; Mosaic data augmentation is turned off in the last 10 epochs to improve the model's adaptability to real-world scenarios.

[0062] The loss function is designed as follows: Classification loss:

[0063] Regression loss:

[0064] Total loss function:

[0065] in, To balance the hyperparameters, the relative weights of classification loss and regression loss are adjusted.

[0066] S3: Integrated OCR Recognition Engine S31: Construct a multi-engine collaborative recognition framework. Integrate multiple OCR recognition engines such as EasyOCR, Tesseract, and PaddleOCR, leverage the advantages of each engine in different scenarios, and obtain diverse recognition candidate results through parallel reasoning.

[0067] S32: Design an adaptive image preprocessing workflow. Based on the characteristics of scale images, generate various preprocessing variants, including CLAHE contrast enhancement, adaptive threshold binarization, Otsu global threshold segmentation, color inversion, morphological closing operation, and sharpening filtering, to provide diverse inputs for OCR recognition and improve the recognition success rate.

[0068] S33: Implement an intelligent voting selection mechanism. Collect the recognition results of all OCR engines on all preprocessed images and establish a voting statistical model for candidate numbers. Calculate a comprehensive score for each candidate number.

[0069] Selecting the candidate with the highest score as the final recognition result effectively suppresses random errors in a single recognition.

[0070] S34: Strict number validity verification is employed. The recognition results are format-checked, and only text that conforms to the number specifications (integers, decimals, and decimals with leading zeros) is accepted. Invalid characters are filtered out to ensure the reliability of the recognition results.

[0071] S4: Precise pointer positioning under geometric constraints: S41: Adaptive Circular Dial Detection. A multi-strategy fusion method for dial localization is employed: Hough circle transform is prioritized for detecting standard circular dials; for elliptical or perspective-distorted dials, edge detection combined with an ellipse fitting algorithm is used; in cases of detection failure, a default estimation strategy based on the image center is used to ensure system robustness.

[0072] S42: Pointer detection within a circular region constraint. Construct a circular region of interest centered on the dial's center, with an inner diameter of 0.2 times the dial radius (to preserve the root of the hand) and an outer diameter of 0.95 times the dial radius (to exclude interference from the dial's edges). Perform pointer detection within this circular region to effectively eliminate interference from the dial's central decoration and edge markings.

[0073] S43: Multi-scale morphological enhancement. CLAHE histogram equalization is applied to the circular region image to enhance contrast, Gaussian blur is used to suppress noise, Otsu thresholding is used to extract the foreground, and morphological closing operations are used to connect broken pointer line segments, creating favorable conditions for subsequent line detection.

[0074] S44: Probabilistic Hough Line Transform Detection. The HoughLinesP algorithm is used to detect pointer line segments. An adaptive minimum line segment length threshold is set (not less than 0.2 times the dial radius), and the segmented pointers are connected by setting a reasonable maximum gap parameter.

[0075] S45: Pointer Candidate Optimization Strategy. For all detected line candidates, calculate their distance from the center of the dial and the length of the line segment. Define a comprehensive scoring function: Score = Line Segment Length × (1 / Distance to Center). Select the line segment with the highest score as the pointer. This scoring function considers both the length characteristics of the pointer and the geometric characteristic that the pointer must be close to the center of the dial.

[0076] S46: Backup contour detection mechanism. When line detection fails, a backup strategy based on connected component analysis is activated: extract the largest connected component within the annular region, calculate the direction angle of its smallest bounding rectangle as the pointer angle, and ensure that the system can still provide reasonable results in extreme cases.

[0077] S5: Angle Calculation and Reading Conversion of Geometric Relationships S51: Establish a standardized angular coordinate system. With the center of the dial as the origin, the vertical downward direction as the 0° reference, and the clockwise direction as the angle growth direction, establish a standard angular coordinate system from 0° to 360°.

[0078] S52: Calculate the absolute angle of the scale markings. Based on the center coordinates of the detection boxes for the minimum and maximum scale values, calculate their angular position relative to the center of the dial to obtain the geometric distribution information of the scale.

[0079] S53: Calculate the absolute angle of the pointer. Based on the coordinates of the endpoints of the pointer line segment, identify the endpoint farther from the center as the pointer tip, and calculate the angle of the pointer tip relative to the center.

[0080] S54: Handling of zero-point crossings in angle difference calculation. Design an angle difference calculation algorithm to correctly handle cases where the angle crosses the 0° / 360° boundary. Calculate the clockwise angle range from the minimum to the maximum scale, and the clockwise angle offset of the pointer relative to the minimum scale.

[0081] S55: Linear interpolation calculation of readings. Based on the angular proportionality, a linear interpolation formula is used to calculate the instrument reading.

[0082] Apply boundary constraints to the calculation results to ensure that the readings are between the minimum and maximum values.

[0083] S6: Visualization and Results Output: S61: Generate multi-level visualization results. Draw detection boxes and label information on the original image, marking the positions of the dashboard, maximum value, and minimum value; draw the center of the circle, the outline of the dashboard, the pointer line segment, and the scale reference line on the cropped dial image; overlay and display the recognized values ​​and calculated angle information.

[0084] S62: Output structured recognition results. Generate a result dictionary containing complete information such as read values, pointer angles, scale ranges, angle ranges, and center coordinates, supporting subsequent data analysis and system integration.

[0085] like Figure 2 As shown, a complete instrument reading recognition process includes the following steps: S1: System initialization phase. Load the pre-trained YOLO multi-object detection model weight file, initialize the integrated OCR recognition engine, including EasyOCR, Tesseract, and other recognizers, and configure relevant runtime environment parameters.

[0086] S2: Image Acquisition and Preprocessing. Read the dashboard image to be recognized from the camera or image file, perform basic size checks and format verification on the image to ensure the validity of the input data.

[0087] S3: Multi-object collaborative detection. The preprocessed image is input into the YOLO detection model, and forward inference is performed to simultaneously detect three types of objects: the dashboard main body (class_id=0), the minimum scale value label (class_id=1), and the maximum scale value label (class_id=2). The bounding box coordinates and confidence scores of each object are obtained.

[0088] S4: Grouping and Association of Detection Results. Spatial association analysis is performed on multiple detected targets. The Euclidean distance between each scale mark and the main body of the dashboard is calculated. The nearest neighbor principle is used to match the scale marks with the corresponding main body of the dashboard, forming a dashboard group (gauge_group) data structure.

[0089] S5: Target Region Cropping. Based on the detection box coordinates, sub-images of the dashboard area, the maximum scale value area, and the minimum scale value area are cropped from the original image to prepare for subsequent fine-tuning.

[0090] S6: Dashboard Circular Detection. The cropped dashboard image is converted to grayscale and Gaussian blurred. First, the Hough circle transform is attempted to detect a standard circular dashboard. A reasonable parameter range is set (dp=1, minDist=image short side / 2, param1=50, param2=30) to detect the center coordinates and radius of the dashboard.

[0091] S7: Backup Ellipse Detection. If Hough circle detection fails, enable edge detection and ellipse fitting strategies, try multiple sets of Canny edge detection parameters (30-100, 50-150, 20-80, 40-120), perform ellipse fitting on the detected contours, and select the optimal ellipse as the dial contour based on feature scores such as area and major-minor axis ratio.

[0092] S8: Default center estimation. If all the above detections fail, the default strategy is adopted, using the image center as the center of the dial and 40% of the shorter side as the radius, ensuring that the system can continue processing even in extreme cases.

[0093] S9: Integrated OCR recognition of scale values. For the cropped maximum and minimum value images, seven preprocessed variants are generated: original magnified image, CLAHE enhanced image, adaptive threshold binary image, Otsu binary image, inverted Otsu image, morphological closing operation image, and sharpened binary clockwise image. For each preprocessed image, EasyOCR and Tesseract are called in parallel for recognition, and all recognition results and their confidence scores are collected.

[0094] S10: Voting to select the optimal result. Statistical analysis is performed on all collected candidate numbers, calculating the frequency of occurrence and average confidence level of each number, and then applying the weighted scoring formula (…). The highest-scoring number is selected as the final recognition result. If OCR recognition fails, the minimum value is set to 0.0 by default, and the maximum value is set to a reasonable default value (such as 25.0) based on experience.

[0095] S11: Calculate the absolute angle of the scale markings. Convert the center coordinates of the minimum and maximum value detection boxes to the original image coordinate system and calculate their angles relative to the center of the dial. Use the atan2 function to calculate the angle and convert it to a standard coordinate system with vertical downwards as 0° and increasing. Fine-tune the angles (minimum angle -3°, maximum angle +3°) to compensate for the offset between the scale marking boxes and the actual scale lines.

[0096] S12: Construct a ring-shaped detection area. Using the detected center of the dial as the center, create a ring-shaped mask with an inner diameter of 0.2 times the radius and an outer diameter of 0.95 times the radius. Perform pointer detection within this area to effectively eliminate interference from the dial's central decoration and edge markings.

[0097] S13: Pointer image preprocessing. Perform CLAHE contrast enhancement (clipLimit=3.0), Gaussian blur noise reduction (kernel=5×5), Otsu threshold binarization, color inversion (making the pointer a white foreground), and morphological closing operation (kernel=3×3) to connect broken line segments on the circular region image.

[0098] S14: Canny edge detection. Perform Canny edge detection (threshold 10-50) on the preprocessed binary image to extract edge information, preparing for Hough line detection.

[0099] S15: Probabilistic Hough Line Detection. The HoughLinesP algorithm is used to detect line segments, with the minimum segment length set to max(15, 0.2 × radius) and the maximum gap set to 5 pixels. All candidate lines within a circular region are detected.

[0100] S16: Pointer candidate selection. For each detected straight line, calculate its length and distance to the center of the circle:

[0101] Select the line with the highest score as the pointer. Determine the distance from each endpoint of the line to the center of the circle; the endpoint farther from the center is the pointer tip.

[0102] S17: Backup contour detection. If line detection fails, perform connected component analysis on the binary image, extract the largest connected component, and calculate the direction angle of its smallest bounding rectangle as the pointer angle.

[0103] S18: Calculate the absolute angle of the pointer. Based on the coordinates of the pointer tip and the center of the circle, use the atan2 function to calculate the absolute angle of the pointer and convert it to the standard coordinate system (0° = vertically downward, increasing clockwise).

[0104] S19: Angle difference calculation. Calculate the clockwise angle range from the smallest to the largest scale (handling cases spanning 360°), and calculate the clockwise angle offset of the pointer relative to the smallest scale.

[0105] S20: Linear interpolation is used to calculate the reading. Based on the angular proportions, the formula is:

[0106] Apply boundary constraints to the calculation results to ensure that the readings are within the range of [minimum value, maximum value].

[0107] S21: Result Output and Visualization. Generate structured results containing information such as reading values, pointer angle, scale range, and center coordinates. Draw visualization information such as detection boxes, center points, pointers, and scale reference lines on the original and cropped images, and save the visualization result images.

[0108] The method provided in this application embodiment can also realize parallel processing of multiple dashboards. For a scene image containing multiple dashboards, the system executes the following processing flow: Step 1: The YOLO model detects all dashboards and scale markings in the image in one pass; Step 2: Based on spatial relationships, the test results are automatically grouped, and each dashboard is paired with its nearest scale mark; Step 3: For each instrument panel group, execute steps S5 to S20 in Example 1 in parallel to achieve simultaneous readings from multiple instruments; Step 4: Summarize the readings from all instruments and generate a batch processing report.

[0109] Through the above technical solution, the present invention realizes fully automatic, high-precision, and robust reading recognition of the instrument panel, which is applicable to various application scenarios such as industrial automation, intelligent inspection, and remote monitoring.

[0110] In summary, the deep learning-based intelligent dashboard detection and reading method provided in this application is highly intelligent and automated. Through multi-target collaborative detection technology, it achieves simultaneous recognition of the dashboard body and scale markings, eliminating the need for manual configuration of parameters such as scale range. The system's automation level reaches an industry-leading level. Compared to traditional methods, deployment efficiency is improved by 80%, and maintenance costs are reduced by 60%.

[0111] It boasts exceptional recognition accuracy, employing an integrated OCR recognition engine and intelligent voting mechanism. Even in challenging scenarios such as complex lighting, small fonts, and low resolution, the scale value recognition accuracy reaches over 98.5%. The geometrically constrained pointer positioning algorithm enables angle measurement accuracy to reach ±0.5°, with reading errors controlled within ±1% of full scale, meeting the demands of high-precision industrial applications.

[0112] With strong environmental adaptability, the system can effectively cope with complex working conditions such as perspective distortion, uneven lighting, blurred pointers, and partial occlusion through innovative designs such as multi-strategy fusion dial detection, diversified image preprocessing, and backup detection mechanisms. The success rate of recognition in actual industrial environments reaches more than 95%, which is significantly better than the 70%-80% success rate of traditional methods.

[0113] It boasts excellent real-time performance, based on a lightweight deep learning model and optimized processing flow. The end-to-end processing time for a single image is less than 200 milliseconds. It supports real-time video stream analysis and can be deployed on edge computing devices to meet the real-time monitoring needs of industrial sites.

[0114] It has broad application prospects, good versatility and scalability. It is not only suitable for conventional instruments such as pressure gauges and thermometers, but can also be extended to various meter types such as electricity meters, water meters, and gas meters. It has broad application prospects in industries such as power, petroleum, chemical, manufacturing, and municipal engineering. It can effectively solve the pain points of low efficiency and easy error in traditional manual meter reading.

[0115] See Figure 3 This application embodiment can also provide a deep learning-based intelligent detection and reading device for dashboards, such as... Figure 3 As shown, the device may include: The multi-target collaborative detection unit 301 is used to construct a multi-target collaborative detection model. It simultaneously detects the dashboard main body, the maximum scale value label, and the minimum scale value label through a single deep learning forward propagation, and obtains the bounding box coordinates of the three types of targets. The final scale value selection unit 302 is used to generate multiple image preprocessing variants for the detected maximum scale value region and minimum scale value region using an integrated OCR recognition engine, and to call multiple OCR engines in parallel for recognition, and select the final scale value through a weighted voting mechanism. The pointer detection unit 303 is used to perform pointer positioning with geometric constraints, construct an annular region of interest with the detected center of the dial circle as the center, detect pointer line segments within the annular region of interest, and select pointer line segments through a scoring function, wherein the scoring function is the line segment length divided by the distance from the pointer line segment to the center of the circle; The reading calculation unit 304 is used to perform angle calculation and reading conversion based on the scale geometry. It calculates the absolute angle of the scale mark according to the center coordinates of the detection frame based on the minimum and maximum scale values. After calculating the absolute angle of the pointer tip and processing the angle difference across zero points, it calculates the instrument reading based on the angle ratio relationship through a linear interpolation formula.

[0116] This application embodiment can also provide a deep learning-based intelligent dashboard detection and reading device, the device including a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of the deep learning-based intelligent detection and reading method for dashboards according to the instructions in the program code.

[0117] like Figure 4As shown in the figure, an intelligent instrument panel detection and reading device based on deep learning provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0118] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0119] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the deep learning-based intelligent detection and reading method for dashboards.

[0120] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions: A multi-target collaborative detection model is constructed, which simultaneously detects the dashboard main body, the maximum scale value label, and the minimum scale value label through a single deep learning forward propagation, and obtains the bounding box coordinates of the three types of targets; An integrated OCR recognition engine is used to generate multiple image preprocessing variants for the detected maximum and minimum scale value regions, and multiple OCR engines are called in parallel for recognition. The final scale value is selected through a weighted voting mechanism. Pointer positioning with geometric constraints is performed by constructing an annular region of interest centered on the detected dial center. Pointer line segments are detected within the annular region of interest, and the pointer line segments are optimized by a scoring function, which is the line segment length divided by the distance from the pointer line segment to the center of the circle. Angle calculation and reading conversion are performed based on the geometric relationship of the scale. The absolute angle of the scale mark is calculated according to the center coordinates of the detection frame based on the minimum and maximum scale values. After calculating the absolute angle of the pointer tip and processing the angle difference across zero points, the instrument reading is obtained by linear interpolation formula based on the angle ratio relationship.

[0121] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0122] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0123] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0124] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the deep learning-based intelligent dashboard detection and reading device in the embodiments of this application. In practical applications, the deep learning-based intelligent dashboard detection and reading device may include devices that... Figure 4 More or fewer components as shown, or combinations of certain components.

[0125] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described deep learning-based intelligent detection and reading method for dashboards.

[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A deep learning-based intelligent detection and reading method for dashboards, characterized in that, include: A multi-target collaborative detection model is constructed, which simultaneously detects the dashboard main body, the maximum scale value label, and the minimum scale value label through a single deep learning forward propagation, and obtains the bounding box coordinates of the three types of targets; An integrated OCR recognition engine is used to generate multiple image preprocessing variants for the detected maximum and minimum scale value regions, and multiple OCR engines are called in parallel for recognition. The final scale value is selected through a weighted voting mechanism. Pointer positioning with geometric constraints is performed by constructing an annular region of interest centered on the detected dial center. Pointer line segments are detected within the annular region of interest, and the pointer line segments are optimized by a scoring function, which is the line segment length divided by the distance from the pointer line segment to the center of the circle. Angle calculation and reading conversion are performed based on the geometric relationship of the scale. The absolute angle of the scale mark is calculated according to the center coordinates of the detection frame based on the minimum and maximum scale values. After calculating the absolute angle of the pointer tip and processing the angle difference across zero points, the instrument reading is obtained by linear interpolation formula based on the angle ratio relationship.

2. The method according to claim 1, characterized in that, The multi-target collaborative detection model includes: DarkNet-53 backbone feature extraction network, and introduces C2f module to replace C3 module; The path aggregation feature pyramid network employs a bidirectional feature fusion mechanism that combines top-down and bottom-up approaches. The decoupled multi-task detection head adopts an anchorless prediction paradigm and eliminates the objectness branch.

3. The method according to claim 1, characterized in that, The training process of the multi-target collaborative detection model includes: A header region annotation strategy is adopted to select and annotate the header region of the scale numbers; A phased fine-tuning training strategy was adopted, freezing the backbone network parameters for the first 20 training cycles, unfreezing the entire network for the next 50 training cycles, and disabling Mosaic data augmentation for the last 10 training cycles. The overall loss function for training is expressed by the following formula: In the formula: Represents classification loss. Indicates regression loss, This represents the balancing hyperparameter.

4. The method according to claim 1, characterized in that, The aforementioned OCR engines include at least EasyOCR, Tesseract, and PaddleOCR; The various image preprocessing variants include at least CLAHE contrast enhancement, adaptive threshold binarization, Otsu global threshold segmentation, color inversion, morphological closing operation, and sharpening filtering.

5. The method according to claim 1, characterized in that, In the weighted voting mechanism, the formula for calculating the comprehensive score of each candidate number is as follows: The candidate number with the highest score is selected as the final recognition result, and the recognition result is validated for digital validity. Only integers, decimals, or decimals with leading zeros are accepted.

6. The method according to claim 1, characterized in that, The construction parameters for the annular area of ​​interest include an inner diameter equal to 0.2 times the dial radius and an outer diameter equal to 0.95 times the dial radius. The detection of the pointer line segment adopts the probabilistic Hough linear transformation, and the minimum line segment threshold is set to be no less than 0.2 times the dial radius; The calculation formula for the scoring function is as follows: Choose the line with the highest score as the pointer.

7. The method according to claim 1, characterized in that, The detection of the center of the dial uses a multi-strategy fusion method: Hough circle transform is preferred for testing standard circular dials; If the Hough circle transform detection fails, an edge detection combined with an ellipse fitting algorithm is used. If ellipse fitting fails, the center of the image will be used as the center of the dial, and 40% of the shorter side of the image will be used as the radius of the dial.

8. The method according to claim 1, characterized in that, The pointer positioning of the geometric constraints also includes a backup detection mechanism, which includes extracting the largest connected component within the annular region and calculating the direction angle of the smallest bounding rectangle of the connected component as the pointer angle when the probabilistic Hough line detection fails to detect a valid pointer line segment.

9. The method according to claim 1, characterized in that, The angle calculation and reading conversion specifically include: Establish a standardized angular coordinate system with the center of the dial as the origin, the vertical downward direction as the 0° reference, the clockwise direction as the direction of angle increase, and the angle range as 0°~360°; Calculate the absolute angle of the scale markings. Based on the center coordinates of the detection boxes for the minimum and maximum scale values, calculate the angular position of the scale relative to the center of the dial to obtain the geometric distribution information of the scale. Calculate the absolute angle of the pointer. Based on the coordinates of the endpoints of the pointer line segment, identify the endpoint farther from the center of the circle as the pointer tip, and calculate the angle of the pointer tip relative to the center of the circle. The algorithm for calculating angle differences across zero points is designed to handle cases where the angle crosses the 0° / 360° boundary. The clockwise angle range from the minimum to the maximum scale is calculated, as well as the clockwise angle offset of the pointer relative to the minimum scale. Based on the angular ratio, a linear interpolation formula is used to calculate the instrument reading; the linear interpolation formula is as follows: Apply boundary constraints to the calculation results to keep them between the minimum and maximum scale values.

10. The method according to claim 1, characterized in that, It also includes visualization and result output steps: Generate multi-level visualization results: draw detection boxes and label information on the original image, and mark the positions of the dashboard, maximum value, and minimum value; draw the center of the circle, the outline of the dashboard, the pointer line segment, and the scale reference line on the cropped dashboard image; and overlay and display the recognized values ​​and calculated angle information. Generate a structured result dictionary, which includes at least the read value, pointer angle, scale range, angle range, and center coordinates.

Images