A cross-scale retrieval algorithm for instrument panel recognition

Through cross-scale search algorithm, combined with diversified datasets and deep learning technology, the identification problem of dashboards in diverse and complex environments is solved, and the dashboard recognition with high accuracy and robustness is achieved, suitable for real-time monitoring and Automation system.

CN118155196BActive Publication Date: 2024-11-05QINGDAO BOSLAN IOT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410195852.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-22
Publication Date
2024-11-05
Estimated Expiration
2044-02-22

AI Technical Summary

Technical Problem

The prior art is difficult to achieve accurate identification and real-time processing in the diversity and complex environments of instrument panels, especially in the fields of vehicle monitoring and industrial automation, where misreading or delays can lead to serious consequences.

Method used

Using a cross-scale search algorithm, the data sets are integrated by building a diverse dashboard identification, combining feature extraction, K-means clustering, global image similarity matching, foreground segmentation and edge detection of deep learning, DBNetPP and SATRN models are integrated. Perform text detection and recognition to achieve robust recognition of the dashboard.

Benefits of technology

It significantly improves the accuracy and efficiency of dashboard recognition, ensures high accuracy and robustness under various environmental conditions, and is suitable for real-time monitoring and automation systems, providing fast response capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118155196B_ABST
    Figure CN118155196B_ABST
Patent Text Reader

Abstract

The present invention provides a cross-scale retrieval algorithm for instrument panel recognition. First, the algorithm adopts a feature extraction method based on a Transformer network, and simplifies feature processing through K-means clustering, which enhances the robustness and accuracy of the algorithm. The algorithm also includes a TopK similarity matching system based on global image features, and a foreground segmentation process focused on the dial area, which together improve the accuracy and efficiency of recognition. In order to further optimize the reading recognition of the instrument panel, the algorithm combines edge detection and graph network technology to accurately locate the pointer and scale. In addition, by combining the DBNetPP and SATRN models, the algorithm effectively handles text detection and recognition tasks. The algorithm also ensures an efficient learning process through reasonable data set partitioning, optimizer selection, and application of loss functions. Finally, the algorithm showed high accuracy in the testing phase, and by analyzing the test results, the model was further optimized to ensure its high performance in practical applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning technology, and in particular relates to a cross-scale retrieval algorithm for instrument panel recognition. Background Art

[0002] In modern automation and intelligent monitoring systems, the automatic recognition and reading of dashboards is an extremely critical but challenging area. Faced with the extremely high diversity of dashboards in design, size, and layout, as well as the constant changes in environmental factors such as lighting, angles, and occlusion, the algorithm needs to be able to not only accurately process and analyze different types of dashboards, but also adapt to various complex environmental conditions. In addition, the automation system has strict requirements for the accuracy and speed of recognition. In areas such as vehicle monitoring and industrial automation that are highly dependent on real-time data, any misreading or delay may lead to adverse consequences.

[0003] To address these issues, the researchers developed a cross-scale retrieval algorithm that aims to improve the accuracy and efficiency of dashboard recognition. The first step of the algorithm is to build a comprehensive and diverse dashboard recognition cross-scale retrieval dataset, which contains dashboard images of various types and styles. Through in-depth analysis of these images, the algorithm can learn how to effectively recognize under different designs and environmental conditions. Then, the algorithm uses robust feature extraction technology to extract key information between different dashboard images and maintain stable performance even in complex environmental conditions.

[0004] In addition, the algorithm combines the matching methods of global images and local features, and greatly improves the accuracy of matching and recognition by calculating the similarity of the entire image and its key parts. The algorithm also includes a series of advanced image processing techniques, such as foreground segmentation and key point detection, focusing on identifying and parsing key information such as pointers, scales and numbers on the dial. In addition, by integrating advanced text detection (such as DBNetPP) and recognition (such as SATRN) technologies, the algorithm can accurately read digital and text information from the dashboard.

[0005] Ultimately, this cross-scale retrieval algorithm not only solves the diversity and complex environment problems in dashboard recognition, but also ensures high accuracy and robustness under various conditions. Its fast and real-time processing capabilities have made the algorithm widely used in automation systems. In scenarios such as vehicle monitoring, industrial automation, and energy management systems that require fast response and high accuracy, the successful implementation of this algorithm marks an important advancement in artificial intelligence in automation technology and paves the way for future intelligent systems and automation solutions. Summary of the invention

[0006] This paper proposes an efficient cross-scale retrieval algorithm for dashboard recognition, which is designed to process and parse diverse and complex dashboard images. It greatly improves the accuracy and efficiency of dashboard recognition by constructing a comprehensive data set, applying robust feature extraction technology, combining global and local similarity matching, as well as foreground segmentation and key point detection. In addition, the integrated advanced text detection and recognition technology and emphasized real-time processing capabilities make the algorithm particularly suitable for real-time monitoring and automation systems, and can operate stably under various environmental conditions, bringing important innovations to the field of automation technology. Specifically, it includes the following steps:

[0007] 1) Construct a cross-scale retrieval dataset for dashboard recognition

[0008] We use Paddle's Industrial Meter Readings dataset. Paddle's Industrial Meter Readings dataset is an image dataset designed for industrial meter reading applications. The dataset includes various types of pressure gauges and digital meter images captured under different conditions, covering different lighting conditions, angles, and meter types, which helps train models to be robust and accurate in real-world scenarios.

[0009] 2) Robust feature extraction of similar dashboard images

[0010] We use the training set as reference images. These images are used to build a feature extraction system based on the Transformer network. The Transformer network is a powerful deep learning model that can capture high-level features based on local regions or "patches" of an image. This patch-based approach allows the model to focus on the details of the image, thereby extracting a rich and descriptive set of features.

[0011] For each reference image, we use the Transformer network to extract its features. These features not only include the overall visual information of the image, but also go deep into small local areas to capture key details and texture information. This processing method is crucial for understanding and representing images as well as complex image recognition and classification tasks.

[0012] Next, we used the K-means clustering algorithm to further process these feature sets. K-means is an effective unsupervised learning algorithm that discovers patterns in data by grouping data points into K different clusters. In this project, K-means is used to cluster similar patch-based features to form N different cluster centers. Each cluster center represents a collection of features that are visually or statistically similar.

[0013] In this way, we can effectively simplify the complex feature space of reference and test images into tractable cluster centers. This not only improves the efficiency of subsequent image analysis and recognition processes, but also provides an intuitive way to understand key features and patterns in images.

[0014] 3) TopK similarity matching based on global images

[0015] In this step, a series of cluster centers are first generated through K-means clustering, representing the main feature patterns in the data set. These cluster centers are then used when selecting the most similar training images. The features of the test image are compared with these centers, and the similarity is evaluated by calculating the cosine distance. In this way, the system can identify the cluster centers that are most similar to the test image in the feature space, and determine the most relevant training images based on this. These selected training images are closest to the test images in terms of global features, providing an ideal basis for subsequent anomaly detection.

[0016] 4) Foreground dial segmentation based on dial positioning area

[0017] Next, the K dashboard images obtained by combining the top K cluster centers are combined for collaborative object segmentation. The main reason is accuracy: by selecting the most similar K dashboard images, the segmentation process is ensured to focus on the samples that are closest to the test image in terms of global features, which increases the accuracy of segmentation. Limiting the segmentation process to the top K images instead of the entire dataset improves processing speed and efficiency. This approach focuses on the most relevant samples and improves the segmentation effect of the test image. The pointer localization method based on foreground dial segmentation provides significant advantages when processing images with complex or changing background environments. First, by focusing on the segmented dial foreground, this method greatly reduces the interference of background noise and irrelevant information, thereby improving the accuracy of pointer detection and localization. Second, the processing speed is optimized because pointer detection in a small and focused area reduces the amount of data that needs to be processed. In addition, this method improves the adaptability and robustness of the algorithm, enabling it to handle dashboards of different types and designs, even if they are in different backgrounds. In addition, this method also improves flexibility, allowing the segmentation and localization algorithms to be individually adjusted for different image types or application requirements. Finally, the clearly segmented dial area provides a simple and clear input for subsequent image analysis tasks, thereby simplifying the entire processing flow.

[0018] 5) Edge detection based on foreground dial segmentation

[0019] On a watch dial, the position of the hands is often closely related to the relative position of the digital markers. Learning edge features in an image using a convolutional neural network (CNN) through a deep learning version of Canny edge detection plays an important role in understanding the spatial and visual relationship between these elements. This approach enables the edge detection network to identify potential hand positions and digital positions, providing key information for accurate hand positioning. Through the combination of deep learning and edge detection, the position and features of these key elements are effectively captured in the dial image, allowing for accurate hand positioning.

[0020] 6) Pointer positioning and scale key point detection based on graph network

[0021] The method of building a graph network and establishing relationships based on the edge information detected by HED brings multiple benefits to instrument panel recognition. First, it accurately locates the boundaries of instrument panel elements, including pointers, scales, and digital marks, improving the accuracy of position and outline. Second, by building a graph network and establishing relationships between elements, it helps to understand the structure of the instrument panel and the interaction between elements. In addition, the graph network is able to capture the contextual information of the elements and the surrounding environment, providing a comprehensive visual understanding. This method supports multi-tasking, allowing multiple instrument panel recognition tasks such as pointer position detection, digital readings, and warning sign recognition to be performed simultaneously, improving efficiency and accuracy. Most importantly, it improves the robustness of the recognition algorithm, reduces the impact of illumination and noise on the recognition results, and is suitable for instrument panel applications in different environments and conditions.

[0022] 7) DBNetPP-based text detection and SATRN-based text recognition use two deep learning models, DBNetPP and SATRN, to achieve efficient image text processing tasks. DBNetPP is used for text detection, which can accurately locate the text area and reduce the risk of misidentification, while SATRN is used for text recognition, which automatically aligns and recognizes text content and improves the accuracy of text recognition. The combination of these two models is suitable for optical character recognition and natural scene text recognition applications, providing a powerful tool for text processing tasks, while accelerating model training and adjustment, making it easy to adapt to different tasks.

[0023] 8) Instrument panel reading recognition

[0024] In this case, calculating the reading of a needle on a dashboard involves first determining the current position of the needle, and then determining the position and angle of the needle relative to the two nearest tick marks based on this position. Specifically, we first find the positions of the two tick marks before and after the needle, and then measure the current angular position of the needle. Then, by calculating the difference in distance between the two tick marks and the relative change in the angle of the needle, we are able to calculate the exact reading of the needle. This process relies heavily on some basic principles of geometry and triangulation.

[0025] 9) Training and testing of a cross-scale retrieval algorithm for instrument panel recognition

[0026] During the training process of this method, the dataset is divided into training set, validation set and test set. This method sets a gradually decreasing learning rate, selects an appropriate batch size, and determines a certain number of training cycles. According to different tasks, this method adopts corresponding loss functions and uses Adam optimizer. In order to improve the generalization ability of the model, this method also applies Dropout and data augmentation techniques. During the testing process, this method loads the trained model, preprocesses the test set images, and performs performance evaluation, and finally optimizes the model based on the test results.

[0027] Through the above steps, a cross-scale retrieval algorithm for dashboard recognition is implemented. Compared with the prior art, the beneficial effects of this method are:

[0028] Enhanced diversity and adaptability to complex environments: This algorithm is specifically designed to handle the high diversity of dashboard designs, sizes, and layouts, as well as the constant changes in lighting, angles, and occlusions. This enables the algorithm to run stably in a variety of complex environments, improving its reliability and applicability in practical applications.

[0029] Significant improvement in accuracy and speed: By combining comprehensive dataset construction, robust feature extraction techniques, and advanced image processing methods, the algorithm significantly improves the accuracy of dashboard recognition. At the same time, its emphasis on real-time processing ensures fast responsiveness, which is particularly important in scenarios that require high accuracy and fast response.

[0030] Accurate dial reading and pointer positioning: The algorithm can accurately read the digital and text information on the dashboard by integrating advanced text detection and recognition technology. In addition, the algorithm also focuses on identifying and parsing key information such as pointers, scales and numbers on the dial, improving the accuracy of reading and the ability to capture details.

[0031] Wide range of application areas: This cross-scale retrieval algorithm not only solves the diversity and complex environment problems in dashboard recognition, but also finds wide application in automation systems due to its high accuracy and robustness. In scenarios such as vehicle monitoring, industrial automation, and energy management systems that require fast response and high accuracy, this algorithm shows great application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structural framework of a cross-scale retrieval algorithm for instrument panel recognition provided by the present invention;

[0033] Figure 2 This is the complete architecture of the network proposed by the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described below through examples in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way. The present invention discloses a cross-scale retrieval algorithm for instrument panel recognition - a cross-scale retrieval algorithm for instrument panel recognition. This algorithm not only shows excellent adaptability and accuracy in the repair process, but also effectively integrates a variety of advanced image processing and machine learning technologies. This innovative technology marks a major breakthrough in the field of instrument panel recognition and lays a solid foundation for realizing efficient and accurate cross-scale retrieval algorithms for instrument panel recognition. Specifically, the algorithm is implemented through the following key steps: The specific implementation includes the following steps:

[0035] 1) Construct a cross-scale retrieval dataset for dashboard recognition

[0036] We use Paddle's Industrial Meter Readings dataset. Paddle's Industrial Meter Readings dataset is an image dataset designed for industrial meter reading applications. The dataset includes various types of pressure gauges and digital meter images captured under different conditions, covering different lighting conditions, angles, and meter types, which helps train models to be robust and accurate in real-world scenarios.

[0037] In order to increase the dashboard recognition dataset by searching images by image, this method selects some representative dashboard images as the search starting point, and then uses image search engines such as Google Images and Bing Images to search. In the search results, suitable and similar dashboard images are screened out, and copyright issues are taken into consideration. After that, the collected images are cleaned and preprocessed to ensure that they meet the format and quality standards of the dataset. Finally, appropriate annotations are given to the new images, such as dial readings and pointer positions, and then they are integrated into the existing dataset.

[0038] 2) Robust feature extraction of similar dashboard images

[0039] We use the training set as reference images. These images are used to build a feature extraction system based on the Transformer network. The Transformer network is a powerful deep learning model that can capture high-level features based on local regions or "patches" of an image. This patch-based approach allows the model to focus on the details of the image, thereby extracting a rich and descriptive set of features.

[0040] For each reference image, we apply the Transformer network to extract its features. These features not only include the overall visual information of the image, but also go deep into small local areas, capturing key details and texture information. This processing is crucial for understanding and representing images and complex image recognition and classification tasks.

[0041] Next, we used the K-means clustering algorithm to further process these feature sets. K-means is an effective unsupervised learning algorithm that discovers patterns in data by grouping data points into K different clusters. In this project, K-means is used to cluster similar patch-based features to form N different cluster centers. Each cluster center represents a collection of features that are visually or statistically similar.

[0042] In this way, we can effectively simplify the complex feature space of reference and test images into tractable cluster centers. This not only improves the efficiency of subsequent image analysis and recognition processes, but also provides an intuitive way to understand key features and patterns in images.

[0043] First, a Transformer network is used to extract feature vectors from each reference image:

[0044] f i = Transformer(I i )

[0045] (f i ) is the feature vector extracted by the Transformer network from a single image i.

[0046] Combine the feature vectors of all reference images into one large feature set:

[0047] F={f1,f2,…,f n}

[0048] Among them, (F) is the set of feature vectors, (f i ) is the feature vector of a single image.

[0049]

[0050] Among them, (F total ) is the merged feature set.

[0051] Perform K-means clustering on the merged feature set:

[0052] K-means(F total )→{C1,C2,…,C k}

[0053] Among them, {C1,C2,…,C k} is the cluster center.

[0054] In this way, K-means clustering helps simplify the processing of image features while retaining the relevance to the original image, facilitating subsequent image retrieval and matching tasks.

[0055] 3) TopK similarity matching based on global images

[0056] In this step, a series of cluster centers are first generated through K-means clustering, representing the main feature patterns in the data set. These cluster centers are then used when selecting the most similar training images. The features of the test image are compared with these centers, and the similarity is evaluated by calculating the cosine distance. In this way, the system can identify the cluster centers that are most similar to the test image in the feature space, and determine the most relevant training images based on this. These selected training images are closest to the test images in terms of global features, providing an ideal basis for subsequent anomaly detection.

[0057] First, get the features of the test image

[0058] f t = Transformer(I t )

[0059] f t is the feature vector of the test image, Transformer represents the model used for feature extraction, and I t is the test image. The next step is to calculate the test image feature vector f t With each cluster center C i The cosine similarity between .

[0060]

[0061] Here, the similarity (f t ,C i ) is used to calculate the test image feature vector ft and cluster center C i The cosine similarity between .

[0062] The next step is to sort the calculated similarities and select the Top K cluster centers with the highest similarity.

[0063] Sort the similarities and select the Top K cluster centers

[0064] Select Top K Centers=Sort(CosineSimilarity(ft ,C1),CosineSimilarity(f t ,C2),…,CosineSimilarity(f t ,C k ))

[0065] Here, the Sort function will sort all cluster centers according to the value calculated by CosineSimilarity and select the Top K centers with the highest similarity. The training images corresponding to these centers will be used for subsequent analysis and processing.

[0066] 4) Foreground dial segmentation based on dial positioning area

[0067] Next, we combine the K dashboard images obtained from the Top K cluster centers for collaborative object segmentation. The main reason is accuracy: by selecting the most similar K dashboard images, we ensure that the segmentation process focuses on the samples that are closest to the test image in terms of global features, which increases the accuracy of segmentation. Limiting the segmentation process to the Top K images instead of the entire dataset improves processing speed and efficiency. This approach focuses on the most relevant samples and improves the segmentation effect of the test image.

[0068] The specific method of performing collaborative object segmentation on the K dashboard images obtained by selecting the Top K cluster centers includes:

[0069] The process of extracting the top K features from the feature set F = f1, f2, ..., fn is expressed by the following formula:

[0070] {f1′,f2′,…,fK′}=TopKFeatures(F,K)

[0071] Among them, F = {f1, f2, ..., fn} is the original feature set, and TopKFeatures (F, K) represents the top K important features selected from F. {f1′, f2′, ..., fK′} are K dashboard images.

[0072] Apply Segmentation Model: Apply the collaborative object segmentation model to these images. The model will perform pixel-level segmentation on each image to identify the exact location and outline of the dashboard. First, perform similarity calculation on the extracted dashboard features. This step is achieved by comparing the similarity of features from different dashboard images to identify areas with high similarity. Convert ft to a column vector Ft. Construct a matrix F′ where each row represents a feature fi′ in the feature set. Calculate the product of F′ and Ft to get the similarity. SimilarityVector=F′×Ft.

[0073] Here, SimilarityVector is a column vector, where each element represents the similarity between Ft and the corresponding feature in the feature set {f1′, f2′, …, fK′}. In this way, the similarity score between ft and each feature in the feature set is obtained.

[0074] This usually indicates the same dashboard section. Next, the identified dashboard features are converted into a foreground region mask. This is done through a decoder network that is able to reconstruct a pixel-level representation of the image based on the features. Finally, this mask is used to segment out the foreground region of the dashboard. In this step, the mask acts as a filter to help determine which pixels belong to the foreground.

[0075] ForegroundMaski=DecodeNetwork(fi′)

[0076] ForegroundMaskt=DecodeNetwork(ft)

[0077] ForegroundMaski represents the foreground instrument panel mask of the i-th image, ForegroundMaskt represents the foreground instrument panel mask of the t-th image, and the decoding network DecodeNetwork is responsible for generating pixel-level foreground area masks based on the instrument panel features. The decoder network mainly includes deconvolution and relu activation layers.

[0078] The pointer localization method based on foreground dial segmentation provides significant advantages when processing images with complex or changing background environments. First, by focusing on the segmented dial foreground, this method greatly reduces the interference of background noise and irrelevant information, thereby improving the accuracy of pointer detection and localization. Second, the processing speed is optimized because pointer detection in a small and focused area reduces the amount of data that needs to be processed. In addition, this method improves the adaptability and robustness of the algorithm, enabling it to handle dashboards of different types and designs, even if they are in different backgrounds. In addition, this method also improves flexibility, allowing the segmentation and localization algorithms to be individually adjusted for different image types or application requirements. Finally, the clearly segmented dial area provides a simple and clear input for subsequent image analysis tasks, thereby simplifying the entire processing pipeline.

[0079] 5) Edge detection based on foreground dial segmentation

[0080] On a watch dial, the position of the hands is often closely related to the relative position of the digital markers. Learning edge features in an image using a convolutional neural network (CNN) through a deep learning version of Canny edge detection plays an important role in understanding the spatial and visual relationship between these elements. This approach enables the edge detection network to identify potential hand positions and digital positions, providing key information for accurate hand positioning. Through the combination of deep learning and edge detection, the position and features of these key elements are effectively captured in the dial image, allowing for accurate hand positioning.

[0081] EdgeMap i =HED(ForegroundMask i )

[0082] EdgeMap t =HED(ForegroundMask t )

[0083] This formula means that ForegroundMask i / ForegroundMask t Input into the Holistically-NestedEdge Detection model and then generate the corresponding edge image EdgeMap i / EdgeMap t , where edges are labeled 1 and non-edges are labeled 0.

[0084] 6) Pointer positioning and scale key point detection based on graph network

[0085] The method of building a graph network and establishing relationships based on the edge information detected by HED brings multiple benefits to instrument panel recognition. First, it accurately locates the boundaries of instrument panel elements, including pointers, scales, and digital marks, improving the accuracy of position and outline. Second, by building a graph network and establishing relationships between elements, it helps to understand the structure of the instrument panel and the interaction between elements. In addition, the graph network is able to capture the contextual information of the elements and the surrounding environment, providing a comprehensive visual understanding. This method supports multi-tasking, allowing multiple instrument panel recognition tasks such as pointer position detection, digital readings, and warning sign recognition to be performed simultaneously, improving efficiency and accuracy. Most importantly, it improves the robustness of the recognition algorithm, reduces the impact of illumination and noise on the recognition results, and is suitable for instrument panel applications in different environments and conditions.

[0086] Define the nodes of the graph as the locations of edge pixels. Each edge pixel becomes a node in the graph. Edges represent the connection relationship between nodes. Based on the relationship between edge pixels, a graph network is constructed. This is achieved by establishing connections between edge pixels in the image. Define a graph network Graph, which contains a node set V and an edge set E. The node set V includes the edge pixel locations in the edge image E. The edge set E represents the edges connecting these nodes, usually expressed as:

[0087] Graph=(V,E)

[0088] Where V = {(x, y)|(x, y) is the edge image EdgeMap i / EdgeMap t edge pixel position in}

[0089] E={(v i ,v j )|v i and v j Is the adjacent EdgeMap i / EdgeMap t Middle edge pixel position}

[0090] Among them, V is the node set, E is the edge set, (x, y) represents the coordinate position of the edge pixel, and there is a connection between adjacent edge pixel positions.

[0091] 7) DBNetPP-based text detection and SATRN-based text recognition use two deep learning models, DBNetPP and SATRN, to achieve efficient image text processing tasks. DBNetPP is used for text detection, which can accurately locate the text area and reduce the risk of misidentification, while SATRN is used for text recognition, which automatically aligns and recognizes text content and improves the accuracy of text recognition. The combination of these two models is suitable for optical character recognition and natural scene text recognition applications, providing a powerful tool for text processing tasks, while accelerating model training and adjustment, making it easy to adapt to different tasks.

[0092] DBNetPP is used for text detection, and its main goal is to detect the location and bounding box of text from an image. DBNetPP is a deep convolutional neural network that extracts features layer by layer through convolution and pooling layers, and then uses multi-layer feature fusion and downsampling methods to detect text regions. It usually uses sliding windows or anchor boxes to generate candidate text regions, and classifies and regresses these regions to obtain the final text detection results.

[0093] DO=DBNetPP(Graph)=(C(si),R(si))∣si∈S(M(F(Graph)))

[0094] DO represents the result of text detection, F represents the feature extraction function, which usually includes convolution layer, activation function and pooling layer. M represents the feature fusion and downsampling function. After feature extraction, the extracted features are usually fused and downsampled by different methods to reduce the dimension of the data and enhance the detection ability of text areas. S represents the candidate text area generation function, which is used to generate candidate text areas. A sliding window is used to identify areas in the image that may contain text. C represents the classification function. For each candidate area, the classification function determines whether this area contains text. R represents the regression function. For areas classified as text, the regression function further determines the specific location of the text and the size of the bounding box.

[0095] SATRN is a text recognition model based on the attention mechanism. It uses a convolutional neural network (CNN) to extract features of text regions, and then uses a self-attention mechanism to recognize text by step-by-step decoding. SATRN usually includes an encoder and a decoder part, where the decoder part uses a self-attention mechanism to focus on features at different positions to generate recognition results. The text recognition results of ATRN can be used to convert characters or words in a text region into readable text to achieve the function of optical character recognition (OCR).

[0096]

[0097] o t Represents the text detection output result, Indicates that the outputs of all time steps are merged into the final text result. Trans converts the output ot of the decoder at each time step into text. A represents the attention mechanism, T: the length of the output sequence, o t : The output of the current time step, D is the decoder.

[0098] 8) Instrument panel reading recognition

[0099] Known o t After that, get the position of the previous scale point (r1), which is the distance of the previous scale that the pointer points to, usually the distance from the origin of the coordinate system to the scale. Get the position of the next scale point (r2), which is the distance of the next scale that the pointer points to, also the distance from the origin of the coordinate system to the scale. Get the current angular position of the pointer (θr), which is the angular position of the pointer in the polar coordinate system, usually expressed in radians.

[0100] Assume that the position of the front scale point of the pointer is r1, and the position of the rear scale point is r2.

[0101] The number is:

[0102] r=r1+δr*δθ,

[0103] δr=r2-r1,

[0104]

[0105] Calculate the scale difference and angle difference: Calculate the scale difference (δr), which is the difference between the position of the front scale point (r1) and the position of the rear scale point (r2), which is used to represent the distance between the two scales, that is, δr = r2-r1,

[0106] Calculate the angle difference (δθ): Use the position of the front scale point (r1), the position of the rear scale point (r2), and the current angle position of the pointer (θr) to calculate according to the following formula:

[0107]

[0108] Among them, θ1 and θ2 are the angular positions of the front scale point and the back scale point respectively.

[0109] Calculate the scale information (r) of the current pointer, use the scale difference (δr) and the angle difference (δθ), and calculate the scale information of the current pointer according to the following formula: r=r1+δr*δθ.

[0110] 9) Training and testing of a cross-scale retrieval algorithm for instrument panel recognition

[0111] Training process: The dataset is divided into training set, validation set and test set. If the ratio is 70% training set, 15% validation set and 15% test set. The initial learning rate is set to 0.001, and the learning rate decay strategy is used, which is halved every 5 epochs. The batch size is set to 32. The number of iterations is set to 50 epochs. For pointer position classification, the cross entropy loss function is used. For pointer angle estimation, the mean square error (MSE) loss function is used. For text detection and recognition, Dice loss is used. Adam is used as the optimizer. Dropout is applied to reduce overfitting, and data augmentation techniques, image rotation, scaling and color adjustment are used to improve the generalization ability of the model. Testing process: Load the trained model. And perform the same preprocessing steps on the images in the test set, such as scaling and normalization. Finally, analyze the model's misrecognition cases on the test set. Further optimize the model based on the error analysis results.

[0112] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A cross-scale retrieval algorithm for dashboard recognition, Its characteristics are The steps include: 1) Construct a cross-scale retrieval dataset for dashboard recognition This method uses Paddle's industrial instrument reading dataset, which is an image dataset designed for industrial instrument reading applications. The dataset includes various types of pressure gauges and digital instrument images captured under different conditions. These images cover different lighting conditions, angles, and instrument types, which helps train the model to have strong robustness and accuracy in real-world scenarios. 2) Robust feature extraction of similar dashboard images The training set is used as reference images. These images are used to build a feature extraction system based on the Transformer network, a powerful deep learning model that can capture high-level features based on local regions or "patches" of images. This patch-based approach allows the model to focus on the details of the image, thereby extracting a rich and descriptive feature set. For each reference image, a Transformer network is used to extract its features. These features not only include the overall visual information of the image, but also go deep into small local areas to capture key details and texture information. This processing method is crucial for understanding and representing images as well as complex image recognition and classification tasks. Next, the K-means clustering algorithm was used to further process these feature sets. K-means is an effective unsupervised learning algorithm that discovers patterns in data by grouping data points into K different clusters. K-means is used to cluster similar patch-based features to form N different cluster centers. Each cluster center represents a set of features that are visually or statistically similar. In this way, the complex feature space of the reference image and the test image can be effectively simplified into easy-to-handle cluster centers, which not only improves the efficiency of the subsequent image analysis and recognition process, but also provides an intuitive way to understand the key features and patterns in the image; 3) TopK similarity matching based on global images In this step, a series of cluster centers are first generated by K-means clustering to represent the main feature patterns in the data set. Then, when selecting the most similar training images, these cluster centers are used to compare the features of the test image with these centers, and the similarity is evaluated by calculating the cosine distance. In this way, the system can identify the cluster center that is most similar to the test image in the feature space, and determine the most relevant training images based on this. These selected training images are closest to the test images in terms of global features, providing an ideal basis for subsequent anomaly detection; 4) Foreground dial segmentation based on dial positioning area Next, the K dashboard images obtained by combining the Top K cluster centers are combined for collaborative object segmentation. The main reason is accuracy: by selecting the most similar K dashboard images, the segmentation process is ensured to focus on the samples that are closest to the test image in global features, which increases the accuracy of segmentation. Limiting the segmentation process to the Top K images instead of the entire data set improves processing speed and efficiency. This method focuses on the most relevant samples and improves the segmentation effect of the test image. The pointer positioning method based on foreground dial segmentation provides significant advantages when processing images with complex or changing background environments. First, by focusing on the segmented dial foreground, this method greatly reduces the interference of background noise and irrelevant information, thereby improving the accuracy of pointer detection and positioning. Secondly, the processing speed is optimized because pointer detection in a small and focused area reduces the amount of data required to be processed. In addition, this method improves the adaptability and robustness of the algorithm, enabling it to handle dashboards of different types and designs, even if they are in different backgrounds. In addition, this method also improves flexibility, allowing the segmentation and positioning algorithms to be adjusted separately for different image types or application requirements. Finally, the clearly segmented dial area provides a simple and clear input for subsequent image analysis tasks, thereby simplifying the entire processing flow. 5) Edge detection based on foreground dial segmentation On a watch dial, the position of the hands is usually closely related to the relative position of the digital markings. Using a deep learning version of Canny edge detection, a convolutional neural network (CNN) is used to learn edge features in an image, which plays an important role in understanding the spatial and visual relationship between these elements. This method enables the edge detection network to identify potential hands and digital positions, providing key information for accurate hands positioning. Through the combination of deep learning and edge detection, the positions and features of these key elements are effectively captured in the dial image, thereby achieving accurate hands positioning. 6) Pointer positioning and scale key point detection based on graph network The method of building a graph network and establishing relationships based on the edge information detected by HED brings multiple benefits to instrument panel recognition. First, it accurately locates the boundaries of instrument panel elements, including pointers, scales, and digital marks, improving the accuracy of position and outline. Second, by building a graph network and establishing relationships between elements, it helps to understand the structure of the instrument panel and the interaction between elements. In addition, the graph network can also capture the contextual information of the elements and the surrounding environment, providing a comprehensive visual understanding. This method supports multi-tasking and allows multiple instrument panel recognition tasks to be performed simultaneously, such as pointer position detection, digital readings, and warning sign recognition, improving efficiency and accuracy. Most importantly, it improves the robustness of the recognition algorithm and reduces the impact of light and noise on the recognition results, making it suitable for instrument panel applications in different environments and conditions. 7) Text detection based on DBNetPP and text recognition based on SATRN model Use DBNetPP and SATRN, two deep learning models, to achieve efficient image text processing tasks. DBNetPP is used for text detection and can accurately locate text areas to reduce the risk of misidentification, while SATRN is used for text recognition and automatically aligns and recognizes text content, improving the accuracy of text recognition. The combination of these two models is suitable for optical character recognition and natural scene text recognition applications, providing a powerful tool for text processing tasks while accelerating model training and adjustment, making it easy to adapt to different tasks; 8) Instrument panel reading recognition In this case, calculating the reading of the pointer on the dashboard involves first determining the current position of the pointer, and then determining the position and angle of the pointer relative to the two nearest scale points based on this position. Specifically, first find the positions of the two scale points before and after the pointer, and then measure the current angular position of the pointer. Then, by calculating the distance difference between the two scale points and the relative change in the pointer angle, the accurate reading of the pointer can be calculated. This process mainly relies on some basic geometric and triangulation principles; 9) Training and testing of a cross-scale retrieval algorithm for instrument panel recognition During the training process, the dataset is divided into training set, validation set and test set. A gradually decreasing learning rate is set, the appropriate batch size is selected, and a certain number of training cycles are determined. According to different tasks, this method adopts the corresponding loss function and uses the Adam optimizer. In order to improve the generalization ability of the model, this method also applies Dropout and data augmentation technology. During the testing process, the trained model is loaded, the test set images are preprocessed, and the performance is evaluated. Finally, the model is optimized according to the test results.

2. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 2) We use the training set as reference images, which are used to build a feature extraction system based on the Transformer network, a powerful deep learning model that can capture high-level features based on local regions or "patches" of images. This patch-based approach allows the model to focus on the details of the image, thereby extracting a rich and descriptive feature set; For each reference image, we use the Transformer network to extract its features. These features not only include the overall visual information of the image, but also go deep into small local areas, capturing key details and texture information. This processing method is crucial for understanding and representing images as well as complex image recognition and classification tasks; Next, we used the K-means clustering algorithm to further process these feature sets. K-means is an effective unsupervised learning algorithm that discovers patterns in data by grouping data points into K different clusters. In this project, K-means is used to cluster similar patch-based features to form N different cluster centers. Each cluster center represents a set of features that are visually or statistically similar. In this way, we can effectively simplify the complex feature space of the reference image and the test image into easy-to-handle cluster centers, which not only improves the efficiency of the subsequent image analysis and recognition process, but also provides an intuitive way to understand the key features and patterns in the image. First, a Transformer network is used to extract feature vectors from each reference image: f i =Transformer(I i ); Single Image I i , Combine the feature vectors of all reference images into one large feature set: F={f 1 ,f 2 ,…,f n }; Among them, (F) is the set of feature vectors, (f i ) is the feature vector of a single image, Among them, (F total ) is the merged feature set, Perform K-means clustering on the merged feature set: K-means(F total )→{C 1 ,C 2 ,…,C k }; Among them, {C 1 ,C 2 ,…,C k } is the cluster center, In this way, K-means clustering helps simplify the processing of image features while retaining the relevance to the original image, facilitating subsequent image retrieval and matching tasks.

3. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 3) In this step, a series of cluster centers are first generated through K-means clustering to represent the main feature patterns in the data set. Then, when selecting the most similar training images, these cluster centers are used to compare the features of the test image with these centers, and the similarity is evaluated by calculating the cosine distance. In this way, the system can identify the cluster center that is most similar to the test image in the feature space, and determine the most relevant training image based on this. These selected training images are closest to the test image in terms of global features, providing an ideal basis for subsequent anomaly detection. First, get the features of the test image f t =Transformer(I t ); f t is the feature vector of the test image, Transformer represents the model used for feature extraction, and I t is the test image, and the next step is to calculate the test image feature vector f t With each cluster center C i The cosine similarity between Here, the similarity (f t , C i ) is used to calculate the test image feature vector ft and cluster center C i The cosine similarity between The next step is to sort the calculated similarities and select the top K cluster centers with the highest similarity. Sort the similarities and select the Top K cluster centers Select Top K Centers=Sort(CosineSimilarity(f t ,C 1 ),CosineSimilarity(f t ,C 2 ),…,CosineSimilarity(f t ,C k ))。 4. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 4) Next, the K dashboard images obtained by combining the Top K cluster centers are combined for collaborative object segmentation. The main reason is accuracy: by selecting the most similar K dashboard images, the segmentation process is ensured to focus on the samples that are closest to the test image in terms of global features, which increases the accuracy of segmentation. Limiting the segmentation process to the Top K images instead of the entire dataset improves processing speed and efficiency. This method focuses on the most relevant samples and improves the segmentation effect of the test image. The specific method of performing collaborative object segmentation on the K dashboard images obtained by selecting the Top K cluster centers includes: The process of extracting the top K features from the feature set F = f1, f2, ..., fn is expressed by the following formula: {f1′, f2′,…,fK′}=TopKFeatures(F,K); Among them, F = {f1, f2, ..., fn} is the original feature set, and TopKFeatures (F, K) represents the top K important features selected from F, and {f1′, f2′, ..., fK′} is K dashboard images. Apply the segmentation model: Apply the collaborative object segmentation model to these images. The model will perform pixel-level segmentation on each image to identify the exact location and outline of the dashboard. First, the extracted dashboard features are similarity calculated. This step is achieved by comparing the similarity of features from different dashboard images to identify areas with high similarity. ft is converted to a column vector Ft, and a matrix F′ is constructed, where each row represents a feature fi′ in the feature set. The product of F′ and Ft is calculated to get the similarity, SimilarityVector = F′×Ft; Here, SimilarityVector is a column vector, in which each element represents the similarity between Ft and the corresponding feature in the feature set {f1′, f2′, …, fK′}. In this way, the similarity score between ft and each feature in the feature set is obtained. This usually indicates the same dashboard part. Next, the identified dashboard features are converted into a foreground region mask. This is achieved through a decoding network that can reconstruct the pixel-level representation of the image based on the features. Finally, this mask is used to segment the foreground region of the dashboard. In this step, the mask acts as a filter to help determine which pixels belong to the foreground. ForegroundMaski=DecodeNetwork(fi′); ForegroundMaskt=DecodeNetwork(ft); ForegroundMaski represents the foreground instrument panel mask of the i-th image, ForegroundMaskt represents the foreground instrument panel mask of the t-th image, and the decoding network DecodeNetwork is responsible for generating pixel-level foreground area masks based on the instrument panel features. The decoder network mainly includes deconvolution and relu activation layers; The pointer localization method based on foreground dial segmentation provides significant advantages when processing images with complex or changing background environments. First, by focusing on the segmented dial foreground, this method greatly reduces the interference of background noise and irrelevant information, thereby improving the accuracy of pointer detection and localization. Second, the processing speed is optimized because pointer detection in a small and focused area reduces the amount of data required to be processed. In addition, this method improves the adaptability and robustness of the algorithm, enabling it to handle dashboards of different types and designs, even if they are in different backgrounds. In addition, this method also improves flexibility, allowing the segmentation and localization algorithms to be individually adjusted for different image types or application requirements. Finally, the clearly segmented dial area provides a simple and clear input for subsequent image analysis tasks, thereby simplifying the entire processing flow.

5. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 5) On the dial, the position of the pointer is usually closely related to the relative position of the digital mark. Through the deep learning version of Canny edge detection, the edge features in the image are learned using a convolutional neural network (CNN), which plays an important role in understanding the spatial and visual relationship between these elements. This method enables the edge detection network to identify potential pointer positions and digital positions, providing key information for accurate pointer positioning. Through the combination of deep learning and edge detection, the position and features of these key elements are effectively captured in the dial image, thereby achieving accurate pointer positioning. EdgeMap i =HED(ForegroundMask i ); EdgeMap t =HED(ForegroundMask t ); This formula means that ForegroundMask i / ForegroundMask t Input into the Holistically-NestedEdge Detection model and then generate the corresponding edge image EdgeMap i / EdgeMap t , where edges are labeled 1 and non-edges are labeled 0.

6. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 6) The method of building a graph network and establishing relationships based on the edge information detected by HED brings multiple benefits to instrument panel recognition. First, it accurately locates the boundaries of instrument panel elements, including pointers, scales, and digital marks, improving the accuracy of position and contour. Second, by building a graph network and establishing relationships between elements, it helps to understand the structure of the instrument panel and the interaction between elements. In addition, the graph network can also capture the contextual information of the elements and the surrounding environment, providing a comprehensive visual understanding. This method supports multi-tasking and allows multiple instrument panel recognition tasks to be performed simultaneously, such as pointer position detection, digital readings, and warning sign recognition, improving efficiency and accuracy. Most importantly, it improves the robustness of the recognition algorithm and reduces the impact of light and noise on the recognition results. It is suitable for instrument panel applications in different environments and conditions. The nodes of the graph are defined as the positions of edge pixels. Each edge pixel will become a node in the graph. The edge represents the connection relationship between nodes. According to the relationship between edge pixels, a graph network is constructed. This is achieved by establishing connections between edge pixels in the image. A graph network Graph is defined, which contains a node set V and an edge set E. The node set V includes the edge pixel positions in the edge image E. The edge set E represents the edges connecting these nodes, which is usually expressed as: Graph=(V,E) Where V = {(x, y)|(x, y) is the edge image EdgeMap i / EdgeMap t edge pixel position in}; E={(v i ,v j )|v i and v j Is the adjacent EdgeMap i / EdgeMap t edge pixel position}; where V is the node set, E is the edge set, (x, y) represents the coordinate position of the edge pixel, and there is a connection between adjacent edge pixel positions.

7. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 7) Use DBNetPP and SATRN, two deep learning models, to achieve efficient image text processing tasks. DBNetPP is used for text detection, which can accurately locate text areas and reduce the risk of misrecognition, while SATRN is used for text recognition, which automatically aligns and recognizes text content and improves the accuracy of text recognition. The combination of these two models is suitable for optical character recognition and natural scene text recognition applications, providing a powerful tool for text processing tasks, while accelerating model training and adjustment, making it easy to adapt to different tasks. DBNetPP is used for text detection. Its main goal is to detect the location and bounding box of text from an image. DBNetPP is a deep convolutional neural network that extracts features layer by layer through convolution and pooling layers, and then uses multi-layer feature fusion and downsampling methods to detect text areas. It usually uses sliding windows or anchor boxes to generate candidate text areas, and classifies and regresses these areas to obtain the final text detection results. DO=DBNetPP(Graph)=(C(s i ),R(s i ))∣si∈S(M(F(Graph))); DO represents the result of text detection, F represents the feature extraction function, which usually includes convolution layer, activation function and pooling layer, M represents the feature fusion and downsampling function. After feature extraction, the extracted features are usually fused and downsampled by different methods to reduce the dimension of the data and enhance the detection ability of text areas. S represents the candidate text area generation function, which is used to generate candidate text areas and use sliding windows to identify areas in the image that may contain text. C represents the classification function. For each candidate area, the classification function determines whether the area contains text. R represents the regression function. For areas classified as text, the regression function further determines the specific location of the text and the size of the bounding box. SATRN is a text recognition model based on the attention mechanism. It uses a convolutional neural network (CNN) to extract the features of the text area, and then uses the self-attention mechanism to recognize the text through step-by-step decoding. SATRN usually includes an encoder and a decoder part, in which the decoder part uses the self-attention mechanism to focus on the features at different positions to generate recognition results. The text recognition results of SATRN can be used to convert characters or words in the text area into readable text to realize the function of optical character recognition (OCR). o t Represents the text detection output result, Indicates that the output of all time steps is merged into the final text result. Trans combines the output of the decoder at each time step into the final text result. t Converted to text, A represents the attention mechanism, T: the length of the output sequence, o t : The output of the current time step, D is the decoder.

8. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 8) Known text detection output result o t Finally, get the position of the front scale point (r1), which is the distance of the previous scale pointed by the pointer, usually the distance from the origin of the coordinate system to the scale, get the position of the back scale point (r2), which is the distance of the next scale pointed by the pointer, also the distance from the origin of the coordinate system to the scale, get the current angular position of the pointer (θ r ), which is the angular position of the pointer in polar coordinates, usually expressed in radians, Assuming the position of the front scale point of the pointer is r1, and the position of the rear scale point is r2, the pointer reading is: r=r1+δr*δθ, δr=r2-r1, Calculate the scale difference and angle difference: Calculate the scale difference (δ r ), which is the difference between the position of the front scale point (r1) and the position of the back scale point (r2), used to represent the distance between the two scales, i.e., δ r =r2-r1, Calculate the angle difference (δθ): Use the position of the front scale point (r1), the position of the back scale point (r2), and the current angular position of the pointer (θ r ), calculated according to the following formula: Among them, θ1 and θ2 are the angular positions of the front scale point and the back scale point respectively. Calculate the current pointer scale information (r), using the scale difference (δ r ) and the angle difference (δθ), calculate the scale information of the current pointer according to the following formula: r=r1+δr*δθ.

9. A cross-scale retrieval algorithm for dashboard recognition as claimed in claim 1, Its characteristics are: Step 9) Training process: The dataset is divided into training set, validation set and test set, with a ratio of 70% training set, 15% validation set and 15% test set. The initial learning rate is set to 0.001, and the learning rate decay strategy is used. It is halved every 5 epochs. The batch size (Batch Size) is set to 32, and the number of iterations (Epochs) is set to 50 epochs. For pointer position classification, the cross entropy loss function is used. For pointer angle estimation, the mean square error (MSE) loss function is used. For text detection and recognition, Dice loss is used. Adam is used as the optimizer, and Dropout is applied to reduce overfitting. Data augmentation technology, image rotation, scaling and color adjustment are used to improve the generalization ability of the model. Testing process: Load the trained model and perform the same preprocessing steps on the images in the test set, such as scaling and normalization. Finally, analyze the model's misrecognition cases on the test set and further optimize the model based on the error analysis results.