Instrument panel artificial intelligence identification method and system based on edge calculation

Abstract

The invention relates to the technical field of artificial intelligence, and particularly discloses an instrument panel artificial intelligence identification method and system based on edge calculation. The method comprises the following steps: collecting an instrument panel image, and constructing a pointer type instrument panel data set; building a deep neural network model, detecting a region containing an instrument panel in the image through the deep neural network model, and cutting out an instrument panel region image; correcting the cut instrument panel area image into an instrument panel area image at a front view angle; based on the corrected instrument panel area image, obtaining a scale segmentation image and a pointer segmentation image of the instrument panel by training an end-to-end deep neural network model; and judging whether the instrument panel is a uniform instrument panel, identifying a scale numerical value by combining an OCR technology, and calculating a pointer reading through an included angle proportion and linear interpolation. According to the method, the optimized model is deployed to the edge end equipment, real-time processing and cloud data feedback are realized, and the recognition precision of the instrument panel with uniform and non-uniform scales is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method and system for artificial intelligence recognition of instrument panels based on edge computing. Background Art

[0002] Pointer instrument panels have extensive and important applications in industry, especially in fields such as factories, nuclear power plants, petrochemicals, and transportation. Pointer instrument panels are used to display key parameters such as temperature, pressure, and flow rate. The readings of these instrument panels are crucial for the safe operation and maintenance of equipment.

[0003] In practical applications, there are different types of pointer instrument panels in factories: digital instrument panels, uniformly scaled instrument panels, and non-uniformly scaled instrument panels. Among them, the automatic recognition of digital instrument panels is the simplest. The content of the instrument panel is mainly composed of text and numbers, and can be realized by detecting and decoding with an optical character recognition network. The uniformly scaled pointer instrument panel is more complex than the pure digital instrument panel, but the spacing between scale lines is equal, and the realization of automatic reading is relatively simple. However, for non-uniformly scaled instrument panels, the spacing between scale lines is unequal, the reading is complex and error-prone, and the current automatic instrument panel inspection technology lacks handling for this situation. Non-uniformly scaled instrument panels are commonly found in special application scenarios such as flow rate measurement, and their scale distribution is designed according to specific measurement requirements.

[0004] The existing artificial intelligence instrument panel recognition work mainly focuses on the automatic recognition of uniformly scaled instrument panels and pure digital readings. Using computer vision and artificial intelligence technologies to process uniformly scaled instrument panels has achieved certain results. However, for non-uniformly scaled instrument panels, due to their irregular scale distribution, it is difficult for traditional methods to accurately recognize and read, which has become a technical difficulty.

[0005] In addition, edge devices usually have low computing power and storage resources. Compared with data centers or cloud servers, there are challenges in processing complex deep learning models. Efficiently running deep learning models on resource-constrained edge devices is a technical difficulty. Summary of the Invention

[0006] One technical problem to be solved by the present invention is: to provide a method and system for artificial intelligence recognition of instrument panels, which can efficiently and accurately recognize the readings of uniformly and non-uniformly scaled pointer instrument panels, and is applicable to edge computing, providing strong support for various environments, industrial automation, and intelligence.

[0007] To solve the above technical problem, one technical solution adopted by the present invention is: a method for artificial intelligence recognition of instrument panels based on edge computing, which includes:

[0008] Step 1, collect instrument panel images and construct a pointer instrument panel data set;

[0009] Step 2, build a deep neural network model, detect the area containing the dashboard in the dashboard image through the deep neural network model, and crop out the dashboard area image;

[0010] Step 3, fit the dashboard area image with an ellipse to obtain the minor axis, major axis and direction of the ellipse of the dashboard. According to the minor axis, major axis and direction of the ellipse, correct the dashboard angle through spatial transformation to obtain spatial correction parameters, and correct the cropped dashboard area image to a dashboard area image with a front view angle;

[0011] Step 4, based on the corrected dashboard area image, train an end-to-end deep neural network model to obtain the scale segmentation image and pointer segmentation image of the dashboard;

[0012] Step 5, determine whether it is a uniform dashboard. If yes, go to Step 6; otherwise, go to Step 7;

[0013] Step 6, obtain the positions and values of the zero scale and its adjacent scales, calculate the ratio of the angle between the pointer and the zero scale to the angle between the zero scale and the adjacent scale, perform linear interpolation, calculate the pointer reading, and go to Step 8;

[0014] Step 7, obtain the positions and values of all scales; calculate and compare the angles between the pointer and the zero scale and the angles between each scale and the zero scale, obtain the scale interval where the pointer is located and the two scales enclosing the pointer, calculate the ratio of the angle between the pointer and the smaller scale to the angle between the larger scale and the smaller scale, perform linear interpolation, calculate the pointer reading, and go to Step 8;

[0015] Step 8, transmit the pointer reading back to the cloud.

[0016] Preferably, the deep neural network model is deployed to an edge device, and the edge device is used to transmit the pointer reading back to the cloud.

[0017] Preferably, in Step 2, based on the built deep neural network model, use the YOLO object detection model algorithm to train on the training dataset of the pointer type dashboard dataset; detect the area containing the dashboard in the dashboard image through the feature layers with different resolutions of the deep neural network, and crop out the dashboard area image through the detected category and region feature frame

[0018] Step 3 includes the following steps:

[0019] Step 3.1, training of the dashboard spatial transformation neural network:

[0020] Based on the training dataset of the pointer type dashboard dataset, fit the dashboard area image with an ellipse through OpenCV For the inner dial area, the minor axis, major axis and direction of the ellipse are obtained by fitting.

[0021] According to the minor axis, major axis and direction of the ellipse, the spatial position feature points (x0, y0) of the inner dial area are extracted. The spatial position feature points (x0, y0) include the center of the ellipse, the endpoints of the major axis and the minor axis of the ellipse.

[0022] Based on the obtained spatial position feature points (x0, y0), the spatial position feature points (x0, y0) from different perspectives are fitted to the spatial position feature points (x, y) in the front view through perspective transformation, and the perspective transformation matrix parameter T of the corrected perspective is obtained.

[0023] The dashboard area image is multiplied by the perspective transformation matrix parameter T for spatial transformation to obtain the dashboard area image I at the front view angle. D ;

[0024] Based on the spatial position feature points (x0, y0), (x, y) of the inner dial area before and after fitting and the perspective transformation matrix parameter T, a training data set for supervised learning is established to train the perspective transformation neural network based on Resent18 to implicitly represent the perspective transformation and predict the corresponding implicitly perspective-transformed dashboard image I in the front view. D ;

[0025] Step 3.2, Dashboard spatial transformation inference:

[0026] Based on the image captured by the pan-tilt camera, the dashboard area image is detected and cropped through Step 2. The inner dial spatial position feature points (x'0, y'0) are extracted by fitting the inner dial of the dashboard through OpenCV ellipse. The spatial position feature points (x'0, y'0) include the center of the circle, the endpoints of the major axis and the minor axis of the ellipse.

[0027] The spatial position feature points (x'0, y'0) and the dashboard area image cropped in Step 2 are used as the input of the spatial transformation parameter prediction network to obtain the dashboard area image I' in the front view after spatial transformation. D ;

[0028] In the said Step 4, based on the dashboard area image I' in the front view after spatial transformation D , using an end-to-end neural network with the Resnet18 structure, the dashboard segmentation network is trained by minimizing the Dice loss function through supervised learning to predict the segmentation binary image of the dashboard scale position area and the segmentation binary image of the dashboard pointer area.

[0029] Generate a binary image for dashboard scale segmentation and a binary image for dashboard pointer segmentation based on a trained neural network.

[0030] Preferably, step 4 further includes the following steps: Based on the binary image of dashboard scale segmentation and the binary image of dashboard pointer segmentation, establish a shallow convolutional neural network classifier, and train the output features of the segmentation network through the cross-entropy loss function to obtain the prediction probabilities of uniform and non-uniform. When the predicted probability of the uniform dashboard category is greater than 0.85, it is determined as a uniform dashboard; otherwise, it is a non-uniform dashboard.

[0031] In step 5, determine whether it is a uniform dashboard according to the prediction probabilities of uniform and non-uniform.

[0032] Preferably, step 6 includes the following steps:

[0033] Obtain the positions of the zero scale and its adjacent numerical scales, and perform OCR text recognition on the digital region image of the dashboard and decode it into the dashboard scale numerical text.

[0034] Calculate the angle θ between the zero scale and the adjacent first numerical scale 1-0 , and the angle θ between the pointer and the zero scale p-0 ;

[0035] Calculate the ratio w of the angle between the pointer and the zero scale and the angle between the zero scale and the adjacent first numerical scale,

[0036] According to the calculated ratio w and the numerical value n0 of the zero scale and the numerical value n1 of the first numerical scale, calculate the pointer reading numerical value n p , and obtain the final result of the uniform scale dashboard: n p = w·n1+(1 - w)·n0;

[0037] Step 7 includes the following steps: Obtain the positions of all numerical scales, and perform OCR text recognition on the digital region image of the dashboard and decode it into the scale numerical text.

[0038] Calculate the angle θ between all numerical scales and the zero scale i-0 , i = 1, 2,..., N, and calculate the angle θ between the pointer and the zero scale p-0 ;

[0039] Compare the angle between the pointer and the zero scale with the angles between other scales and the zero scale to obtain the scale interval where the pointer is located, including obtaining the specific numerical values of the two numerical scales that enclose the pointer and their angles with the zero scale

[0040] Calculate the angle between the pointer and the smaller scale The included angle between the larger scale and the smaller scale The ratio w of

[0041] According to the calculated ratio w and the value of the smaller scale And the value of the larger scale Calculate the pointer reading value n p , and obtain the final result of the non-uniform scale dashboard:

[0042] To solve the above technical problems, another technical solution adopted by the present invention is: an edge computing-based dashboard artificial intelligence recognition system, which is characterized in that it includes: an edge device, a pan-tilt camera, a dashboard intelligent recognition module, a TCP / IP communication module, a cloud server and an interaction interface;

[0043] The dashboard intelligent recognition module is a trained deep neural network model deployed on the edge device, and is used to perform dashboard recognition and reading on the dashboard image to obtain dashboard detection information and reading information;

[0044] The edge device is connected to the pan-tilt camera and the TCP / IP communication module;

[0045] The pan-tilt camera can obtain the dashboard image in real time for the dashboard intelligent recognition module to perform dashboard recognition and reading;

[0046] The TCP / IP communication module realizes communication with the cloud interface of the cloud server through the TCP / IP protocol, and transmits the dashboard detection information and reading information back to the cloud server;

[0047] The interaction interface displays the detected dashboard and the dashboard reading in real time.

[0048] Preferably, the edge computing-based dashboard artificial intelligence recognition system further includes: an inspection robot, and the edge device is carried on the inspection robot for real-time monitoring of the factory dashboard through the edge device.

[0049] Preferably, the dashboard intelligent recognition module includes: a dashboard detection module, a dashboard space correction module, a dashboard space segmentation module, a dashboard type discrimination module, and a dashboard pointer reading recognition module;

[0050] The dashboard detection module is used to detect the area containing the dashboard in the dashboard image based on the established deep neural network model, and crop out the dashboard area image;

[0051] The dashboard space correction module is used to fit the dashboard area image by ellipse fitting to obtain the minor axis, major axis and direction of the ellipse, correct the dashboard angle through spatial transformation according to the minor axis, major axis and direction of the ellipse to obtain spatial correction parameters, and correct the cropped dashboard area image into a dashboard area image with a front view angle;

[0052] The dashboard space segmentation module is used to obtain the scale segmentation image and pointer segmentation image of the dashboard based on the corrected dashboard area image by training an end-to-end deep neural network model;

[0053] The dashboard type discrimination module is used to judge whether it is a uniform dashboard;

[0054] The dashboard pointer reading recognition module is used to: when it is a uniform dashboard, obtain the positions and values of the zero scale and its adjacent scales, calculate the ratio of the angle between the pointer and the zero scale to the angle between the zero scale and the adjacent scale, perform linear interpolation, and calculate the pointer reading; when it is a non-uniform dashboard, obtain the positions and values of all scales, calculate and compare the angle between the pointer and the zero scale with the angles between each scale and the zero scale, obtain the scale interval where the pointer is located and the two scales enclosing the pointer, calculate the ratio of the angle between the pointer and the smaller scale to the angle between the larger scale and the smaller scale, perform linear interpolation, and calculate the pointer reading.

[0055] Preferably, the dashboard detection module is used to train on the training dataset of the pointer-type dashboard dataset using the YOLO object detection model algorithm, detect the area containing the dashboard in the dashboard image through the feature layers with different resolutions of the deep neural network, and crop out the dashboard area image through the detected category and the regional feature box

[0056] The dashboard detection module is also used to detect the image captured by the pan-tilt camera using the YOLO object detection model algorithm and crop it based on the regional feature box to obtain the dashboard area image

[0057] The dashboard space correction module is used to fit the inner dashboard area of the dashboard area image by OpenCV ellipse fitting to obtain the minor axis, major axis and direction of the ellipse; according to the minor axis, major axis and direction of the ellipse, extract the spatial position feature points (x0, y0) of the inner dashboard area, and the spatial position feature points (x0, y0) include the center of the ellipse, the endpoints of the major axis and the minor axis of the ellipse; based on the obtained spatial position feature points (x0, y0), fit the spatial position feature points (x0, y0) from different perspectives to the spatial position feature points (x, y) in the front view through perspective transformation to obtain the perspective transformation matrix parameter T for the corrected perspective; the dashboard area image Perform a spatial transformation by multiplying with the perspective transformation matrix parameter T to obtain the image I of the dashboard area at the front view angle D ; Based on the spatial position feature points (x0, y0), (x, y) of the inner dashboard area before and after fitting and the perspective transformation matrix parameter T, establish a training dataset for supervised learning, and train a perspective transformation neural network based on Resent18 to implicitly represent the perspective transformation, and predict the corresponding implicitly perspective-transformed dashboard image I in the front view D ;

[0058] The dashboard space correction module is also used to detect and crop the dashboard area image based on the image captured by the pan-tilt camera through the dashboard detection module Extract the spatial position feature points (x'0, y'0) of the inner dashboard by fitting the inner dashboard of the dashboard with an ellipse using OpenCV. The spatial position feature points (x'0, y'0) include the center of the circle, the endpoints of the major axis and the minor axis of the ellipse; Use the spatial position feature points (x'0, y'0) and the dashboard area image As the input of the spatial transformation parameter prediction network, obtain the dashboard area image I' in the front view after spatial transformation D ;

[0059] The dashboard space segmentation module is used to based on the dashboard area image I' in the front view after spatial transformation D , use an end-to-end neural network with the Resnet18 structure, train the dashboard segmentation network by minimizing the Dice loss function through supervised learning, and predict the segmentation binary image of the dashboard scale position area and the segmentation binary image of the dashboard pointer area; Based on the trained neural network, generate the dashboard scale segmentation binary map and the dashboard pointer segmentation binary map

[0060] Preferably, the dashboard pointer reading recognition module is used to: when the dashboard is uniform, obtain the positions of the zero scale and its adjacent numerical scales, perform OCR text recognition on the digital area image and decode it into the dashboard scale numerical text; calculate the angle θ between the zero scale and the adjacent first numerical scale 1-0 , the angle θ between the pointer and the zero scale p-0 ; Calculate the ratio w of the angle between the pointer and the zero angle and the angle between the zero scale and the adjacent first numerical scale According to the calculated ratio w and the value n0 of the zero scale and the value n1 of the first numerical scale, calculate the pointer reading value n p , obtain the final result of the uniform scale dashboard: n p = w·n1+(1 - w)·n0;

[0061] The dashboard pointer reading recognition module is used for: when the dashboard has non-uniform scales, obtaining the positions of all numerical scales, performing OCR character recognition on the digital area image of the dashboard and decoding it into scale value text; calculating the angle θ between all numerical scales and the zero scale i-0 , where i = 1, 2,..., N, calculating the pointer-zero scale angle θ p-0 ; comparing the pointer-zero scale angle with the angles between other scales and the zero scale to obtain the scale interval where the pointer is located, including obtaining the specific values of the two numerical scales that enclose the pointer and their angles with the zero scale Calculating the angle between the pointer and the smaller scale and the angle between the larger scale and the smaller scale to obtain the ratio w Based on the calculated ratio w and the value of the smaller scale and the value of the larger scale calculate the pointer reading value n p , obtaining the final result of the non-uniform scale dashboard:

[0062] The present invention has the following beneficial effects:

[0063] The artificial intelligence recognition method and system of the present invention can efficiently and accurately automatically recognize and read various types of dashboards, including pointer-type dashboards with uniform and non-uniform scales, improving the level of industrial automation, reducing manual intervention, and lowering the error rate.

[0064] The artificial intelligence recognition method and system of the present invention can achieve real-time dashboard reading in complex environments, ensuring the safe operation of equipment.

[0065] The artificial intelligence recognition method and system of the present invention can adapt to different types of dashboards, have wide applicability, and save development and maintenance costs.

[0066] The artificial intelligence recognition method and system of the present invention can adapt to edge computing and can operate efficiently and accurately with lower computing power and storage resources. Description of the Drawings

[0067] Figure 1 is a flowchart of the artificial intelligence recognition method for dashboards based on edge computing according to an embodiment of the present invention.

[0068] Figure 2 is a schematic structural diagram of the artificial intelligence recognition system for dashboards based on edge computing according to another embodiment of the present invention.

[0069] Figure 3 is a schematic diagram of model conversion compression and deployment according to another embodiment of the present invention. Detailed implementation manners

[0070] The detailed description and technical content of the present invention are described below in conjunction with the accompanying drawings. However, the accompanying drawings are only provided for reference and illustration purposes and are not used to limit the present invention.

[0071] As Figure 1 shown, an edge-computing-based instrument panel artificial intelligence recognition method according to an embodiment of the present invention includes:

[0072] Step 1: Collect instrument panel images and construct a pointer-type instrument panel data set;

[0073] Step 2: Build a deep neural network model, detect the area containing the instrument panel in the instrument panel image through the deep neural network model, and crop out the instrument panel area image, that is, crop out the instrument panel area image based on the detected area;

[0074] Step 3: Fit an ellipse to the instrument panel area image to obtain the minor axis, major axis, and direction of the ellipse of the instrument panel. According to the minor axis, major axis, and direction of the ellipse, correct the instrument panel angle through spatial transformation to obtain spatial correction parameters, and correct the cropped instrument panel area image to an instrument panel area image with a front view angle;

[0075] Step 4: Based on the corrected instrument panel area image, obtain the scale segmentation image and pointer segmentation image of the instrument panel by training an end-to-end deep neural network model;

[0076] Step 5: Determine whether it is a uniform instrument panel. If yes, go to Step 6; otherwise, go to Step 8;

[0077] Step 6: Obtain the positions and values of the zero scale and its adjacent scales, calculate the ratio of the angle between the pointer and the zero scale to the angle between the zero scale and the adjacent scale, perform linear interpolation, calculate the pointer reading, and go to Step 8;

[0078] Step 7: Obtain the positions and values of all scales; calculate and compare the angles between the pointer and the zero scale and the angles between each scale and the zero scale to obtain the scale interval where the pointer is located and the two scales enclosing the pointer. Calculate the ratio of the angle between the pointer and the smaller scale to the angle between the larger scale and the smaller scale, perform linear interpolation, calculate the pointer reading, and go to Step 8;

[0079] Step 8: Transmit the pointer reading back to the cloud.

[0080] Wherein, the deep neural network model is deployed to an edge device, and the edge device is used to transmit the pointer reading back to the cloud.

[0081] In step 1, a pointer-type instrument panel dataset is constructed from the publicly available industrial instrument panel dataset MC1260 and the collected instrument panel data video set. The collection process includes cropping picture frames containing the instrument panel from the videos. Among them, the flow rate instrument panel includes some cases with non-uniform scales. The collected instrument panel data and videos cover different angles and lighting conditions. The pointer-type instrument panel dataset is divided into a training dataset and a test dataset.

[0082] In step 1, it also includes annotating the instrument panel scale values, pointers, pointer readings, and instrument panel types in the instrument panel images, including: annotating the region segmentation binary map of the instrument panel scale values, annotating the region segmentation binary map of the instrument panel pointers, annotating the instrument panel scale values, annotating the readings of the instrument panel pointers, and annotating the instrument panel type as a uniform instrument panel or a non-uniform instrument panel.

[0083] In step 2, based on the constructed deep neural network model, the YOLO object detection model algorithm is used to train on the training dataset of the multi-view and multi-lighting pointer-type instrument panel dataset containing uniform and non-uniform instrument panels constructed in step 1. YOLO(·) is an open-source object detection model; the region containing the instrument panel in the instrument panel image I is detected through the feature layers of different resolutions of the deep neural network. Through the detected category and the region feature box {x min ,y min ,weight,height}, Crop(·) is an operation to crop the image based on the region feature box, and the instrument panel region image is cropped based on the region feature box of the detected instrument panel category.

[0084] Among them,

[0085] {x min ,y min ,weight,height} = YOLO(I), Crop(I, {x min ,y min ,height,weight}).

[0086] For the case of the instrument panel, the angle transformation of the instrument panel is a perspective transformation in three-dimensional space. Therefore, the transformation of the instrument panel scale position at different angles can also be represented by a perspective matrix, and the transformation of the instrument panel scale to the front view angle instrument panel scale is also a perspective transformation, without losing the original scale information.

[0087] In step 3, based on the principle of perspective transformation, perspective transformation processing is performed on the instrument panel image, including central translation, rotation and scaling, and transformation processing of the perspective relationship. The perspective transformation is implicitly expressed by a spatial transformation neural network under supervised learning training. This part is divided into the spatial transformation neural network training part and the spatial transformation inference part based on the trained neural network.

[0088] Step 3 specifically includes the following steps:

[0089] Step 3.1, training of the dashboard space transformation neural network:

[0090] Based on the training dataset, use OpenCV to fit the elliptical inner dashboard area image of the dashboard area image to obtain the minor axis, major axis and direction of the ellipse;

[0091] According to the minor axis, major axis and direction of the ellipse, extract the spatial position feature points (x0, y0) of the inner dashboard area, where the spatial position feature points include the center of the ellipse, the endpoints of the major axis and the minor axis of the ellipse;

[0092] Based on the obtained spatial position feature points, use perspective transformation to fit the feature points from different perspectives to the spatial position feature points (x, y) in the front view, and obtain the perspective transformation matrix parameter T for correcting the perspective, where a ij is the element in the i-th row and j-th column of the perspective transformation matrix, i, j = 1, 2, 3,

[0093]

[0094] Multiply the dashboard area image by the perspective transformation matrix parameter T for spatial transformation, and correct the multi-perspective dashboard image to the dashboard area image I at the front view angle D , (u, v, 1) = (u0, v 0, 1)·T, where (u, v) ∈ I D is the pixel coordinate of the target image I D ; is the image pixel coordinate;

[0095] Based on the spatial position feature points (x0, y0), (x, y) of the inner dashboard area before and after fitting and the perspective transformation matrix parameter T, establish a training dataset for supervised learning, and train the perspective transformation neural network based on Resent18 to implicitly express the perspective transformation. The perspective transformation neural network model is Conv align (·), predict the corresponding implicitly perspective-transformed dashboard area image I in the front view from the image feature points (x0, y0) and the original image (dashboard area image ), D ,

[0096]

[0097] Step 3.2, dashboard space transformation inference:

[0098] The dashboard area image is detected and cropped from the image captured by the pan-tilt camera through Step 2. For the image pixel coordinates, the spatial position feature points (x'0, y'0) of the inner dial are extracted by ellipse fitting of the inner dial in the dashboard using OpenCV, where the spatial position feature points (x'0, y'0) include the center of the circle, the endpoints of the major axis and the minor axis of the ellipse.

[0099] The spatial position feature points (x'0, y'0) and the dashboard area image cropped in Step 2 are used as the input of the spatial transformation parameter prediction network Conv align (·) after training in Step 3.1, and the dashboard area image I' in the front view after spatial transformation is obtained. D , (u', v') ∈ I' D For the image I' D pixel coordinates.

[0100] Step 4 specifically includes the following steps:

[0101] Based on the dashboard area image I' corrected by perspective transformation in Step 3 D , using an end-to-end neural network with the Resnet18 structure, the dashboard segmentation network Resnet is trained by minimizing the Dice loss function through supervised learning seg , and the binary segmentation image of the dashboard scale position area and the binary segmentation image of the dashboard pointer area are predicted;

[0102] Among them, the Dice loss functions for the pointer and the scale are respectively and

[0103]

[0104] Among them, is the binary segmentation map of the pointer predicted by the model, is the real binary segmentation map of the pointer, where the pixels in the pointer area are marked as 1 and the pixels in the non-pointer area are marked as 0; is the binary segmentation map of the dashboard scale predicted by the model, is the real binary segmentation map of the dashboard scale, where the pixels in the scale area are marked as 1 and the pixels in the non-scale area are marked as 0.

[0105] Based on the trained neural network, generate the binary segmentation map of the dashboard scale and the binary segmentation map of the dashboard pointer:

[0106] {M pm , M ks} = Resnet seg (I).

[0107] Step 4 further includes the following steps: Based on the binary space segmented images (i.e., the dashboard scale segmented binary image and the dashboard pointer segmented binary image), a shallow convolutional neural network classifier MLP(·) is established, and through the cross-entropy loss function Train the output features of the segmentation network to obtain the prediction probabilities of uniform and non-uniform. When the predicted probability of the uniform dashboard category is greater than 0.85, it is determined as a uniform dashboard; otherwise, it is a non-uniform dashboard.

[0108]

[0109] where N is the number of samples, p i = MLP(M ks ) is the probability that the classifier predicts the result as the uniform dashboard category, y i is the sign function of the sample. It takes the value of 0 or 1 according to whether it is a uniform scale. If the true category of sample i is uniform, then y i takes 1; otherwise, it takes 0. The dashboard type classifier is trained by minimizing the loss function.

[0110] In step 5, it is judged whether it is a uniform dashboard according to the prediction probabilities of uniform and non-uniform.

[0111] In the present invention, the OpenCV EAST text detector is used to detect the text bounding box (ROI) closest to the numerical scale in the image. Then, the image is denoised and binarized to obtain the character features, and the optical character recognition OCR is used to perform interpretable dashboard reading recognition on the character features within the ROI.

[0112] OCR optical character recognition in the dashboard:

[0113] In OCR optical character recognition, the digital sequence is usually a two-dimensional string feature sequence with an uncertain length. Therefore, after the sequence is filled to a unified length, each individual character is input into the long short-term memory recurrent neural network LSTM for prediction. For the processing of the two-dimensional feature string sequence, all individual two-dimensional character features F ∈ R C×H×W are arranged into a two-dimensional sequence L ∈ R C×W . For each time step t = 0, 1,..., T + 1, l1, l2,..., l ω ∈ L are input into the LSTM to obtain the output y t , and y t belongs to the output space:

[0114] h t ' = f(l t , h t-1 '),

[0115]

[0116] where f(·) is, h t is the hidden layer at time t, and W0 linearly transforms the hidden layer into the output space, the output space y t has a length of 12 and contains a total of 10 Arabic numerals, one '.' character token, and one special termination token;

[0117] Among them, the recognition of a single character is a multi-class classification task based on an image, and the classes are Arabic numerals. The OCR convolutional neural network is trained using the cross-entropy loss function, and the loss function L num is as follows:

[0118]

[0119] where N is the number of samples in the character region of the image, and y n is the sequence recognized by the model, is the true image character sequence.

[0120] If the dashboard is recognized as a uniformly scaled dashboard, according to the situation of the uniform dashboard, since all the scales in the uniform dashboard are evenly distributed, the pointer reading can be obtained by linear interpolation between the first and second numerical scales, and step 6 is executed based on the pointer reading recognition steps for the uniformly scaled dashboard.

[0121] Step 6 specifically includes the following steps:

[0122] Obtain the positions of the zero scale and its adjacent numerical scales, and perform OCR text recognition on the digital region image of the dashboard and decode it into the dashboard scale numerical text;

[0123] Calculate the angle θ 1-0 between the zero scale and the adjacent first numerical scale, and the angle θ p-0 between the pointer and the zero scale;

[0124] Calculate the ratio w of the angle between the pointer and the zero scale and the angle between the zero scale and the adjacent first numerical scale,

[0125] According to the calculated ratio w and the value n0 of the zero scale and the value n1 of the first numerical scale, calculate the pointer reading value n p to obtain the final result of the uniformly scaled dashboard: n p = w·n1 + (1 - w)·n0.

[0126] If the dashboard is recognized as a non-uniform scale dashboard, according to the non-uniform dashboard situation, the numerical scales are non-uniformly distributed, while the scales between two adjacent numerical scales are uniformly distributed. By performing linear interpolation on the two adjacent numerical scales that enclose the pointer, the pointer reading can be predicted. Based on the pointer reading recognition steps for non-uniform scale dashboards, step 7 is executed.

[0127] Step 7 specifically includes the following steps: Obtain the positions of all numerical scales, perform OCR text recognition on the digital area image of the dashboard and decode it into scale value text;

[0128] Calculate the angle θ between all numerical scales and the zero scale i-0 , i = 1, 2,..., N, calculate the pointer-zero scale angle θ p-0 ;

[0129] Compare the pointer-zero scale angle with the angles between other scales and the zero scale to obtain the scale interval where the pointer is located, including obtaining the specific values of the two numerical scales that enclose the pointer and their angles with the zero scale

[0130] Calculate the angle between the pointer and the smaller scale and the angle between the larger scale and the smaller scale ratio w,

[0131] According to the calculated ratio w and the value of the smaller scale and the value of the larger scale Calculate the pointer reading value n p , obtain the final result of the non-uniform scale dashboard:

[0132] In step 8, convert the deep neural network model to a TensorRT compressed half-precision model and deploy it to the edge device. The edge device transmits the pointer reading back to the cloud in real time and displays the reading on the interactive interface in real time.

[0133] In step 8, the returned result is the finally predicted pointer reading value n p , and the marked dashboard position area ROI, the dashboard scale pointer position segmentation map M pm , M ks , the reference numerical scale position position.

[0134] Such as Figure 2As shown in the figure, another embodiment of the present invention, an artificial intelligence recognition system for instrument panels based on edge computing, includes: edge devices, pan-tilt cameras, instrument panel intelligent recognition modules, TCP / IP communication modules, cloud servers, interaction interfaces, and inspection robots.

[0135] The instrument panel intelligent recognition module is a trained deep neural network model deployed on the edge device, and is used to perform instrument panel recognition and reading on the instrument panel image to obtain instrument panel detection information and reading information.

[0136] The instrument panel intelligent recognition module includes: an instrument panel detection module, an instrument panel space correction module, an instrument panel space segmentation module, an instrument panel type discrimination module, and an instrument panel pointer reading recognition module.

[0137] The instrument panel detection module is used to detect the area containing the instrument panel in the instrument panel image based on the established deep neural network model, and crop out the instrument panel area image.

[0138] The instrument panel space correction module is used to fit the instrument panel area image by an ellipse to obtain the minor axis, major axis and direction of the ellipse, correct the instrument panel angle through spatial transformation according to the minor axis, major axis and direction of the ellipse to obtain spatial correction parameters, and correct the cropped instrument panel area image to an instrument panel area image with a front view angle.

[0139] The instrument panel space segmentation module is used to obtain the scale segmentation image and pointer segmentation image of the instrument panel based on the corrected instrument panel area image by training an end-to-end deep neural network model.

[0140] The instrument panel type discrimination module is used to judge whether it is a uniform instrument panel.

[0141] The instrument panel pointer reading recognition module is used for: when it is a uniform instrument panel, obtaining the positions and values of the zero scale and its adjacent scales, calculating the ratio of the angle between the pointer and the zero scale to the angle between the zero scale and the adjacent scale, performing linear interpolation, and calculating the pointer reading; when it is a non-uniform instrument panel, obtaining the positions and values of all scales, calculating and comparing the angle between the pointer and the zero scale with the angles between each scale and the zero scale, obtaining the scale interval where the pointer is located and the two scales enclosing the pointer, calculating the ratio of the angle between the pointer and the smaller scale to the angle between the larger scale and the smaller scale, performing linear interpolation, and calculating the pointer reading.

[0142] The instrument panel pointer reading recognition module is also used to detect the text bounding box (ROI) closest to the numerical scale in the image by using the OpenCV EAST text detector, then denoise and binarize the picture to obtain character features, and perform interpretable instrument panel reading recognition on the character features within the ROI using optical character recognition OCR.

[0143] In a specific embodiment, the dashboard detection module is used to train on the training dataset of the pointer-type dashboard dataset using the YOLO object detection model algorithm. It detects the area containing the dashboard in the dashboard image through the feature layers of different resolutions of the deep neural network, and crops out the dashboard area image based on the detected category and the area feature box.

[0144] The dashboard detection module is also used to detect the image captured by the pan-tilt camera using the YOLO object detection model algorithm and crop it based on the area feature box to obtain the dashboard area image.

[0145] In a specific embodiment, the dashboard space correction module is used to fit the inner dashboard area of the dashboard area image through OpenCV ellipse fitting. The short axis, long axis, and direction of the ellipse are obtained by fitting. According to the short axis, long axis, and direction of the ellipse, the spatial position feature points (x0, y0) of the inner dashboard area are extracted. The spatial position feature points (x0, y0) include the center of the ellipse, the endpoints of the long axis, and the endpoints of the short axis of the ellipse. Based on the obtained spatial position feature points (x0, y0), the spatial position feature points (x0, y0) at different perspectives are fitted to the spatial position feature points (x, y) in the front view through perspective transformation to obtain the perspective transformation matrix parameter T for the corrected perspective. The dashboard area image is multiplied by the perspective transformation matrix parameter T for spatial transformation to obtain the dashboard area image I at the front view angle. D ; Based on the spatial position feature points (x0, y0), (x, y) of the inner dashboard area before and after fitting and the perspective transformation matrix parameter T, a training dataset for supervised learning is established to train the perspective transformation neural network based on Resent18 to implicitly represent the perspective transformation and predict the corresponding implicitly perspective-transformed dashboard image I in the front view. D ;

[0146] The dashboard space correction module is also used to detect and crop the dashboard area image through the dashboard detection module based on the image captured by the pan-tilt camera. The spatial position feature points (x'0, y'0) of the inner dashboard are extracted by fitting the inner dashboard of the dashboard through OpenCV ellipse fitting. The spatial position feature points (x'0, y'0) include the center of the circle, the endpoints of the long axis, and the endpoints of the short axis of the ellipse. The spatial position feature points (x'0, y'0) and the dashboard area image are used as the input of the spatial transformation parameter prediction network to obtain the dashboard area image I' in the front view after spatial transformation. D .

[0147] In a specific embodiment, the dashboard space segmentation module is used to, based on the dashboard area image I' in the front view after spatial transformation D , use an end-to-end neural network with the Resnet18 structure, train the dashboard segmentation network by minimizing the Dice loss function through supervised learning, and predict the segmentation binary image of the dashboard scale position area and the segmentation binary image of the dashboard pointer area; based on the trained neural network, generate the dashboard scale segmentation binary map and the dashboard pointer segmentation binary map.

[0148] In a specific embodiment, the dashboard type discrimination module is used to, based on the binary space segmentation images (i.e., the dashboard scale segmentation binary map and the dashboard pointer segmentation binary map), establish a shallow convolutional neural network classifier MLP(·), and through the cross-entropy loss function train the output features of the segmentation network to obtain the prediction probability of uniform or non-uniform. When the predicted probability of the uniform dashboard category is greater than 0.85, it is determined as a uniform dashboard, otherwise it is a non-uniform dashboard.

[0149]

[0150] where N is the number of samples, p i = MLP(M ks ) is the probability that the classifier predicts the result as the uniform dashboard category, y i is the sign function of the sample. It takes the value of 0 or 1 according to whether it is a uniform scale. If the true category of sample i is uniform, then y i takes 1, otherwise it takes 0; judge whether it is a uniform dashboard according to the prediction probability of uniform or non-uniform.

[0151] In a specific embodiment, the dashboard pointer reading recognition module is used to: when it is a uniform dashboard, obtain the positions of the zero scale and its adjacent numerical scales, perform OCR text recognition on the digital area image and decode it into the dashboard scale numerical text; calculate the angle θ 1-0 between the zero scale and the adjacent first numerical scale, and the angle θ p-0 between the pointer and the zero scale; calculate the ratio w of the angle between the pointer and the zero angle and the angle between the zero scale and the adjacent first numerical scale. According to the calculated ratio w and the value n0 of the zero scale and the value n1 of the first numerical scale, calculate the pointer reading value n p , and obtain the final result of the uniform scale dashboard: n p = w·n1+(1 - w)·n0;

[0152] The dashboard pointer reading recognition module is used for: when the dashboard has non-uniform scales, obtaining the positions of all numerical scales, performing OCR text recognition on the digital area image of the dashboard and decoding it into scale value text; calculating the angle θ between all numerical scales and the zero scale i-0 , where i = 1, 2,..., N, calculating the pointer-zero scale angle θ p-0 ; comparing the pointer-zero scale angle with the angles between other scales and the zero scale to obtain the scale interval where the pointer is located, including obtaining the specific values of the two numerical scales that enclose the pointer and their angles with the zero scale Calculating the angle between the pointer and the smaller scale and the angle between the larger scale and the smaller scale to obtain the ratio w According to the calculated ratio w and the value of the smaller scale and the value of the larger scale calculate the pointer reading value n p , obtaining the final result of the non-uniform scale dashboard:

[0153] As Figure 3 shown, convert the Pytorch file of the trained deep neural network model, compress the model and deploy it to the edge device. Among them, convert the Pytorch file into an ONNX format model file, and then convert it to TensorRT for quantization compression.

[0154] ONNX is an open file format designed for machine learning, used to store trained model data. It enables different artificial intelligence frameworks to store and interact in the same format.

[0155] If the model is to be deployed on Nvidia's edge device, the ONNX model will be converted into a TensorRT model and deployed to the edge device. And perform half-precision compression on the TensorRT model parameters, converting the model parameters from FP32 (floating point 32) to FP16 (floating point 16). TensorRT is a high-performance deep learning inference optimizer that can provide low-latency and high-throughput deployment optimization for generation on Nvidia devices.

[0156] The edge device is connected to a pan-tilt camera and a TCP / IP communication module.

[0157] The pan-tilt camera can obtain the dashboard image in real time for the dashboard intelligent recognition module to perform dashboard recognition and reading.

[0158] The TCP / IP communication module realizes communication with the cloud interface of the cloud server through the TCP / IP protocol, transmits the dashboard detection information and reading information back to the cloud server, supports the real-time feedback of the detection data of the edge device, and ensures the real-time and effectiveness of the detection of the pointer-type dashboard reading by the edge device. When the dashboard is detected and the dashboard reading is successfully detected, the TCP / IP communication module nests the detected dashboard data and the dashboard reading value into the HTTP request POST request and sends it to the cloud server along with the HTTP request. The server side will return an indication of whether the reception is successful. In a specific HTTP request, this project designs the data protocol in JSON format and defines different fields in the protocol to represent different meanings of the data.

[0159] The edge device is carried on the inspection robot and is used to monitor the factory dashboard in real time through the edge device. The edge device uploads the dashboard detection information and reading information to the cloud server in real time and displays the detected dashboard and the dashboard reading on the interactive interface in real time, so as to facilitate the real-time supervision of the operation of the device.

[0160] The above-described embodiments are only preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention.

Claims

1. An artificial intelligence recognition method for instrument panels based on edge computing, characterized in that, It includes: Step 1: Collect dashboard images and construct a pointer-type dashboard dataset; Step 2: Build a deep neural network model, detect the area containing the dashboard in the dashboard image through the deep neural network model, and crop out the dashboard area image; Step 3: Fit an ellipse to the dashboard area image to obtain the minor axis, major axis, and orientation of the ellipse. According to the minor axis, major axis, and orientation of the ellipse, correct the dashboard angle through spatial transformation to obtain spatial correction parameters, and correct the cropped dashboard area image to a dashboard area image with a front view angle; Step 4: Based on the corrected dashboard area image, obtain the scale segmentation image and pointer segmentation image of the dashboard by training an end-to-end deep neural network model; Step 5: Determine whether it is a uniform dashboard. If yes, go to Step 6; otherwise, go to Step 7; Step 6: Obtain the positions and values of the zero scale and its adjacent scales, calculate the ratio of the angle between the pointer and the zero scale to the angle between the zero scale and the adjacent scale, perform linear interpolation, calculate the pointer reading, and go to Step 8; Step 7: Obtain the positions and values of all scales; calculate and compare the angles between the pointer and the zero scale and the angles between each scale and the zero scale to obtain the scale interval where the pointer is located and the two scales that enclose the pointer. Calculate the ratio of the angle between the pointer and the smaller scale to the angle between the larger scale and the smaller scale, perform linear interpolation, calculate the pointer reading, and go to Step 8; Step 8: Transmit the pointer reading back to the cloud.

2. The artificial intelligence recognition method for dashboard based on edge computing according to claim 1, characterized in that, Among them, Deploy the deep neural network model to the edge device, and the edge device is used to transmit the pointer reading back to the cloud.

3. The method for artificial intelligence recognition of a dashboard based on edge computing according to claim 1 or 2, characterized in that In step 2, based on the established deep neural network model, the YOLO object detection model algorithm is used to train on the training dataset of the pointer-type instrument panel dataset; the area containing the instrument panel in the instrument panel image is detected through the feature layers of different resolutions of the deep neural network, and the instrument panel area image is cropped out through the detected categories and regional feature frames Step 3 includes the following steps: Step 3.1: Training of the dashboard spatial transformation neural network: Based on the training dataset of the pointer-type instrument panel dataset, the instrument panel area image is ellipse-fitted by OpenCV for the inner instrument panel area, and the minor axis, major axis, and ellipse direction of the ellipse are obtained by fitting. According to the minor axis, major axis, and orientation of the ellipse, extract the spatial position feature points (x0, y0) of the inner dashboard area. The spatial position feature points (x0, y0) include the center of the ellipse, the endpoints of the major axis, and the endpoints of the minor axis of the ellipse; Based on the obtained spatial position feature points (x0, y0), fit the spatial position feature points (x0, y0) from different perspectives to the spatial position feature points (x, y) in the front view through perspective transformation to obtain the perspective transformation matrix parameter T for the corrected perspective; Multiply the dashboard area image by the perspective transformation matrix parameter T for spatial transformation to obtain the dashboard area image I at the front view angle D ; Based on the spatial position feature points (x0, y0), (x, y) of the inner dial area before and after fitting and the perspective transformation matrix parameter T, a training dataset for supervised learning is established to train a perspective transformation neural network based on Resent18 to implicitly represent the perspective transformation and predict the corresponding dashboard image I in the front view after the implicit perspective transformation D ; Step 3.2: Inference of the dashboard spatial transformation: The dashboard area image is detected and cropped from the image captured by the pan-tilt camera through Step 2 The spatial position feature points (x'0, y'0) of the inner dial are extracted by ellipse fitting of the inner dial in the dashboard using OpenCV. The spatial position feature points (x'0, y'0) include the center of the circle, the endpoints of the major axis and the minor axis of the ellipse; Take the spatial position feature points (x'0, y'0) and the dashboard area image cropped in Step 2 as the input of the spatial transformation parameter prediction network trained in Step 3.1 to obtain the dashboard area image I' in the front view after spatial transformation D ; In the step 4, based on the dashboard area image I' under the front view after spatial transformation D , an end-to-end neural network with Resnet18 structure is used to train the dashboard segmentation network by minimizing the Dice loss function through supervised learning, and a binary segmentation image of the dashboard scale position area and a binary segmentation image of the dashboard pointer area are predicted; Based on the trained neural network, generate a binary map of dashboard scale segmentation and a binary map of dashboard pointer segmentation.

4. The method for artificial intelligence recognition of a dashboard based on edge computing according to claim 3, characterized in that Step 4 further includes the following steps: Based on the binary map of dashboard scale segmentation and the binary map of dashboard pointer segmentation, establish a shallow convolutional neural network classifier, train the output features of the segmentation network through the cross-entropy loss function to obtain the prediction probabilities of uniform and non-uniform. When the predicted probability of the uniform dashboard category is greater than 0.85, it is determined as a uniform dashboard; otherwise, it is a non-uniform dashboard; In Step 5, determine whether it is a uniform dashboard according to the prediction probabilities of uniform and non-uniform.

5. The artificial intelligence recognition method for dashboard based on edge computing according to claim 1 or 2, characterized in that Step 6 includes the following steps: Obtain the positions of the zero scale and its adjacent numerical scales, and perform dashboard OCR text recognition on the digital area image and decode it into dashboard scale numerical text; Calculate the angle θ between the zero scale and the adjacent first numerical scale 1-0 , the angle θ between the pointer and the zero scale p-0 ; Calculate the ratio w of the angle between the calculation pointer and the zero scale to the angle between the zero scale and the adjacent first numerical scale. Calculate the pointer reading value n based on the calculated ratio w and the values n0 of the zero scale and n1 of the first numerical scale p , to obtain the final result of the evenly scaled instrument panel: n p = w·n1 + (1 - w)·n0; Step 7 includes the following steps: Obtain the positions of all numerical scales, perform dashboard OCR text recognition on the digital area image and decode it into scale numerical text; Calculate the angle θ between all numerical scales and the zero scale i-0 , where i = 1, 2, ..., N, calculate the pointer-zero scale angle θ p-0 ; Compare the angle between the pointer and the zero scale with the angles between other scales and the zero scale to obtain the scale interval where the pointer is located, including obtaining the specific numerical values of the two numerical scales that enclose the pointer 0≤k i ≤N - 1, and their angles with the zero scale Calculate the angle between the pointer and the smaller scale and the ratio w of the angle to the angle between the larger scale and the smaller scale According to the calculated ratio w and the value of the smaller scale and the value of the larger scale calculate the pointer reading value n p , to obtain the final result of the non-uniform scale dashboard:

6. An instrument panel artificial intelligence recognition system based on edge computing, characterized in that, It includes: Edge device, pan-tilt camera, dashboard intelligent recognition module, TCP / IP communication module, cloud server and interaction interface; The dashboard intelligent recognition module is a trained deep neural network model deployed on the edge device, and is used to perform dashboard recognition and reading on the dashboard image to obtain dashboard detection information and reading information; The edge device is connected to the pan-tilt camera and the TCP / IP communication module; The pan-tilt camera can obtain the dashboard image in real time for the dashboard intelligent recognition module to perform dashboard recognition and reading; The TCP / IP communication module realizes communication with the cloud interface of the cloud server through the TCP / IP protocol, and transmits the dashboard detection information and reading information back to the cloud server; The interaction interface displays the detected dashboard and the dashboard reading in real time.

7. The dashboard artificial intelligence recognition system based on edge computing according to claim 6, characterized in that, It further includes: an inspection robot, and the edge device is carried on the inspection robot for real-time monitoring of the factory dashboard through the edge device.

8. The artificial intelligence recognition system for dashboard based on edge computing according to claim 6 or 7, characterized in that The dashboard intelligent recognition module includes: a dashboard detection module, a dashboard space correction module, a dashboard space segmentation module, a dashboard type discrimination module, and a dashboard pointer reading recognition module; The dashboard detection module is used to detect the area containing the dashboard in the dashboard image based on the established deep neural network model, and crop out the dashboard area image; The dashboard space correction module is used to fit the dashboard area image by an ellipse to obtain the minor axis, major axis and direction of the ellipse, correct the dashboard angle through space transformation according to the minor axis, major axis and direction of the ellipse to obtain space correction parameters, and correct the cropped dashboard area image into a dashboard area image with a front view angle; The dashboard space segmentation module is used to obtain the scale segmentation image and the pointer segmentation image of the dashboard based on the corrected dashboard area image through training an end-to-end deep neural network model; The dashboard type discrimination module is used to judge whether it is a uniform dashboard; The dashboard pointer reading recognition module is used for: when the dashboard is uniform, obtaining the positions and values of the zero scale and its adjacent scales, calculating the ratio of the angle between the pointer and the zero scale to the angle between the zero scale and the adjacent scale, performing linear interpolation, and calculating the pointer reading; when the dashboard is non-uniform, obtaining the positions and values of all scales, calculating and comparing the angle between the pointer and the zero scale with the angles between each scale and the zero scale, obtaining the scale interval where the pointer is located and the two scales that enclose the pointer, calculating the ratio of the angle between the pointer and the smaller scale to the angle between the larger scale and the smaller scale, performing linear interpolation, and calculating the pointer reading.

9. The artificial intelligence recognition system for dashboard based on edge computing according to claim 8, wherein The dashboard detection module is used to train on the training dataset of the pointer-type dashboard dataset using the YOLO object detection model algorithm. It detects the area containing the dashboard in the dashboard image through the feature layers of different resolutions of the deep neural network, and crops out the dashboard area image based on the detected category and the regional feature box. The dashboard detection module is also used to detect the images captured by the pan-tilt camera using the YOLO object detection model algorithm and crop them based on the regional feature frames to obtain the dashboard area images The dashboard space correction module is used to fit the inner dashboard area image of the dashboard area through OpenCV ellipse fitting, and obtain the minor axis, major axis and direction of the ellipse; according to the minor axis, major axis and direction of the ellipse, extract the spatial position feature points (x0, y0) of the inner dashboard area, and the spatial position feature points (x0, y0) include the center of the ellipse, the endpoints of the major axis and the minor axis of the ellipse; based on the obtained spatial position feature points (x0, y0), fit the spatial position feature points (x0, y0) from different perspectives to the spatial position feature points (x, y) in the front view through perspective transformation, and obtain the perspective transformation matrix parameter T of the corrected perspective; multiply the dashboard area image by the perspective transformation matrix parameter T for spatial transformation to obtain the dashboard area image I at the front view angle ; based on the spatial position feature points (x0, y0), (x, y) of the inner dashboard area before and after fitting and the perspective transformation matrix parameter T, establish a training data set for supervised learning, train the perspective transformation neural network based on Resent18 to implicitly express the perspective transformation, and predict the corresponding implicitly perspective-transformed dashboard image I in the front view D ; D ; The dashboard space correction module is also used to detect and crop the dashboard area image through the dashboard detection module based on the image captured by the pan-tilt camera Extract the spatial position feature points (x'0, y'0) of the inner dial inside the dashboard by ellipse fitting of the inner dial in the dashboard through OpenCV. The spatial position feature points (x'0, y'0) include the center of the circle, the endpoints of the major axis and the minor axis of the ellipse; Use the spatial position feature points (x'0, y'0) and the dashboard area image As the input of the spatial transformation parameter prediction network, obtain the dashboard area image I' in the front view after spatial transformation D ; The dashboard space segmentation module is used for the dashboard area image I' in the front view after space transformation D , using an end-to-end neural network with the Resnet18 structure, training the dashboard segmentation network by minimizing the Dice loss function through supervised learning, and predicting the segmented binary image of the dashboard scale position area and the segmented binary image of the dashboard pointer area; based on the trained neural network, generating the dashboard scale segmentation binary map and the dashboard pointer segmentation binary map.

10. The artificial intelligence recognition system for dashboard based on edge computing according to claim 8, wherein The dashboard pointer reading recognition module is used to: when the dashboard is uniform, obtain the positions of the zero scale and its adjacent numerical scales, perform OCR character recognition on the digital area image of the dashboard, and decode it into the dashboard scale numerical text; calculate the angle θ between the zero scale and the adjacent first numerical scale 1-0 , the angle θ between the pointer and the zero scale p-0 ; calculate the ratio w of the angle between the pointer and the zero angle and the angle between the zero scale and the adjacent first numerical scale According to the calculated ratio w and the value n0 of the zero scale and the value n1 of the first numerical scale, calculate the pointer reading value n p , and obtain the final result of the uniform scale dashboard: n p = w·n1 + (1 - w)·n0; The dashboard pointer reading recognition module is used to: when the dashboard has non-uniform scales, obtain the positions of all numerical scales, perform OCR text recognition on the digital area image of the dashboard, and decode it into scale value text; calculate the angle θ between all numerical scales and the zero scale i-0 , where i = 1, 2,..., N, calculate the pointer-zero scale angle θ p-0 ; compare the pointer-zero scale angle with the angles between other scales and the zero scale to obtain the scale interval where the pointer is located, including obtaining the specific numerical values of the two numerical scales that enclose the pointer 0 ≤ k i ≤ N - 1, and their angles with the zero scale Calculate the angle between the pointer and the smaller scale and the angle between the larger scale and the smaller scale to obtain the ratio w Based on the calculated ratio w and the value of the smaller scale and the value of the larger scale calculate the pointer reading value n p , to obtain the final result of the non-uniform scale dashboard:

Images