Instrument panel and signboard multi-target detection and identification method and system
By using the YOLO multi-target detection model and TensorRT model compression technology in an edge computing environment, the problems of insufficient single-target detection, resource waste, and data privacy in dashboard and signboard multi-target recognition systems are solved, achieving efficient and accurate multi-target recognition and real-time monitoring.
Patent Information
- Application Number
- CN202511076617.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-01
- Publication Date
- 2025-11-18
AI Technical Summary
Existing multi-target detection and recognition systems for dashboards and signs suffer from insufficient functionality of single-target detection models, waste of resources during continuous operation, lack of anomaly detection, and data privacy and security issues, making it difficult to achieve efficient and accurate multi-target recognition and real-time processing in complex scenarios.
A YOLO-based multi-target detection model is adopted, combined with edge computing, to achieve simultaneous detection and recognition of dashboards and signs. The model operation is controlled by monitoring algorithms, and anomaly detection algorithms are introduced. The model is compressed into a TensorRT model and deployed on edge devices to ensure data security.
It achieves efficient and accurate multi-target recognition of dashboards and signs, saves computing resources, provides real-time monitoring and anomaly detection, and ensures data security.
Smart Images
Figure CN120976936A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of dashboard and signage detection and recognition technology, and in particular to a method and system for multi-target detection and recognition of dashboards and signs. Background Technology
[0002] Instrument panels and signage play an indispensable role in many industries, including aerospace, shipbuilding, environmental monitoring, and energy management. Typically, analog instrument panels display important operating parameters in real time, such as current, air pressure, and speed; signage provides equipment information and specific operating instructions.
[0003] For example, in industrial cooling systems, operators need to understand the meaning of the readings on the circulating water pump's pressure gauge by referring to the label text. This allows them to grasp the current outlet pressure of the circulating water pump and take necessary countermeasures when the outlet pressure is abnormal. By recognizing the gauge readings and label text, operators can quickly and accurately understand the equipment status and take appropriate action. Therefore, a multi-target monitoring and identification mechanism for gauge readings and label text is crucial to ensuring the stable operation of the system and the safety of operators.
[0004] Initially, multi-target detection and recognition mechanisms for dashboards and signs in various fields mainly relied on human eye recognition. Traditional human eye recognition methods are not only prone to subjective biases of monitors, resulting in low recognition accuracy and high human resource consumption, but also have low automation levels and are difficult to adapt to complex application scenarios.
[0005] Recently, many recognition systems employing computer vision technology have been proposed. These systems overcome the shortcomings of traditional human eye recognition methods and improve recognition accuracy and efficiency to some extent. However, in multi-target monitoring and recognition scenarios involving dashboards and signs, previous recognition systems still have the following limitations:
[0006] First, the previous recognition system only loaded a single target detection model, which meant that the system could only recognize dashboards or signs and could not complete multi-target recognition tasks.
[0007] Secondly, the previous identification system lacked monitoring capabilities and could not determine whether the target detection model should be started (or stopped) under different circumstances. This resulted in the system needing to run the target detection model continuously for a long time, causing the target detection model in the system to run frequently and ineffectively, wasting a lot of computing resources.
[0008] In addition, the previous recognition system did not have the function of analyzing dashboard readings, making it difficult to provide early warning information for abnormal data and failing to meet the needs of anomaly detection in complex scenarios.
[0009] Finally, because the previous recognition systems were designed with reference to cloud computing architecture rather than edge computing architecture, they not only had difficulty processing dashboard and sign data in real time, but also relied on the security measures of cloud service providers, making it difficult to protect the data privacy of dashboards and signs in complex application scenarios. Summary of the Invention
[0010] The technical problem to be solved by this invention is to provide an innovative method and system for the detection and recognition of dashboards and signs, which enables efficient and accurate dashboard readings and sign text recognition in edge devices, while also enabling process monitoring and anomaly detection for multi-target recognition, providing reliable support for industrial automation and intelligence.
[0011] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a multi-target detection and recognition method for dashboards and signs, comprising:
[0012] Acquire the image to be detected;
[0013] The image to be detected is input into a pre-trained multi-object detection model. The multi-object detection model detects dashboards and / or signs in the image to be detected and segments dashboard area images and / or sign area images. The multi-object detection model uses a YOLO-based deep neural network.
[0014] Recognize dashboard readings in the dashboard area image and / or signage text in the signage area image, and output the recognition results.
[0015] In a preferred embodiment, the method further includes: a step of training a multi-object detection model, specifically including:
[0016] A multi-target recognition dataset is obtained, which includes multiple sample images containing dashboards and signs, as well as annotation information for the sample images. The annotation information includes annotations for dashboard areas, dashboard scale values, dashboard pointers, dashboard pointer readings, dashboard types, and sign areas. The multi-target recognition dataset is divided into a training dataset and a test dataset.
[0017] The training dataset is input into a pre-defined multi-object detection model for training, resulting in a trained multi-object detection model.
[0018] In a preferred embodiment, the multi-target detection model includes: a backbone network, a neck network, and a head network;
[0019] The backbone network uses a convolutional neural network to extract features of the dashboard and signage from the input image and outputs a multi-scale feature map.
[0020] The neck network uses a feature pyramid network to fuse multi-scale features extracted from the backbone network to generate an enhanced feature map.
[0021] The head network makes predictions based on the fused enhanced feature map, outputting the bounding box coordinates, confidence level, and class probability of the target, and segments the dashboard area and / or signage area according to the target's bounding box.
[0022] In a preferred embodiment, after the steps of outputting the bounding box coordinates, confidence level, and class probability of the target, the method further includes: using a non-maximum suppression algorithm to retain the bounding box with the highest confidence level and removing overlapping bounding boxes.
[0023] In a preferred embodiment, the step of identifying dashboard readings in the dashboard area image and / or signage text in the signage area image, and outputting the identification result, includes:
[0024] Ellipse fitting is performed on the dashboard area image to generate the center position of the ellipse, the length of the major axis and the length of the minor axis, as well as the rotation angle of the ellipse.
[0025] After obtaining the ellipse fitting result, a spatial transformation is performed on the ellipse fitting result to correct the dashboard area image to a dashboard area image under a positive viewing angle.
[0026] The nearest numerical scale surrounding the instrument pointer is read from the instrument panel area image under normal viewing angle using optical character recognition. The instrument panel reading is then calculated using linear interpolation and the angle ratio.
[0027] In a preferred embodiment, the step of identifying dashboard readings in the dashboard area image and / or signage text in the signage area image, and outputting the identification result, includes:
[0028] The extracted signage area image is converted to grayscale to generate a grayscale image;
[0029] Define a threshold for binarization, and perform binarization on the grayscale image based on this threshold to remove background noise;
[0030] The binarized image is processed using optical character recognition (OCR) to extract the text from the sign.
[0031] In a preferred embodiment, the step of acquiring the image to be detected further includes a step of monitoring whether one or more images to be detected are being transmitted to a specified path:
[0032] Define a specified path for receiving the image to be detected;
[0033] Define a mask to represent the image creation event in the specified path;
[0034] Create a monitoring list, add the specified path to the monitoring list, and apply the mask;
[0035] Create a monitor to manage the specified paths in the monitoring list;
[0036] Create event handlers to process and dispatch file system events;
[0037] When the monitor detects an image creation event in the specified path, it executes the step of acquiring the image to be detected; when the monitor does not detect an image creation event in the specified path, it does not execute the step of acquiring the image to be detected.
[0038] In a preferred embodiment, the step of identifying dashboard readings in the dashboard area image and / or signage text in the signage area image, and then outputting the identification result, includes the following:
[0039] The system compares the instrument panel readings in the identified instrument panel area image with the preset instrument panel reading thresholds. If not, a warning message is issued.
[0040] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is: a multi-target detection and recognition system for dashboards and signs, configured to implement any of the methods described in the previous technical solution, the system comprising:
[0041] The acquisition module is used to acquire the image to be detected;
[0042] The model processing module is used to input the image to be detected into a pre-trained multi-object detection model. The multi-object detection model detects dashboards and / or signs in the image to be detected and segments the dashboard area image and / or sign area image. The multi-object detection model adopts a YOLO-based deep neural network.
[0043] The recognition module is used to recognize dashboard readings in the dashboard area image and / or signage text in the signage area image, and output the recognition results.
[0044] In a preferred embodiment, the system further includes an edge device, wherein the multi-target detection model is compressed into a half-precision model, converted into a TensorRT model, and then deployed on the edge device.
[0045] The present invention has the following beneficial effects:
[0046] This invention deploys the optimized model to edge devices to achieve real-time monitoring and recognition of multiple targets and cloud data feedback, significantly improving the accuracy of multi-target recognition of dashboard readings and sign text.
[0047] This invention not only has important theoretical significance in the field of dashboard and sign recognition, such as proposing a new recognition framework, but also has wide application value, such as improving the multi-target detection and recognition mechanism, enhancing technical efficiency, and ensuring data security. Attached Figure Description
[0048] Figure 1 This is a flowchart of a multi-target detection and recognition method for dashboards and signs based on edge computing, provided in an embodiment of the present invention.
[0049] Figure 2 This is a schematic diagram of the structure of a multi-target detection and recognition system for dashboards and signs based on edge computing, provided in an embodiment of the present invention.
[0050] Figure 3 This is a schematic flowchart of a multi-target detection and recognition method for dashboards and signs, provided as another embodiment of the present invention. Detailed Implementation
[0051] The detailed description and technical content of the present invention are explained below with reference to the accompanying drawings. However, the drawings are provided for reference and illustration only and are not intended to limit the present invention.
[0052] This invention proposes a multi-target detection and recognition method and system for dashboards and signs based on edge computing, which integrates the following four measures to address four types of shortcomings in previous recognition systems:
[0053] First, a multi-target detection model was built, enabling the model to simultaneously recognize dashboards and signs, thus solving the problem of insufficient functionality of single-target detection models.
[0054] Secondly, a control module based on a monitoring algorithm is constructed to monitor whether new data enters the target folder or other specified paths. If so, the multi-target detection model is started and run; otherwise, the multi-target detection model is terminated and the identified data is cleared, saving a lot of computing resources and storage space.
[0055] In addition, an anomaly detection algorithm is introduced to analyze whether the identified dashboard readings are abnormal. If so, an early warning message is issued to meet the anomaly detection requirements in multi-target monitoring and identification scenarios of dashboards and signs.
[0056] Finally, by employing model optimization strategies such as model compression and model conversion, the multi-target detection model is compressed into a half-precision model, converted into a TensorRT model, and then deployed on edge devices. This enables the system to process data accurately and efficiently in real time, while ensuring data security.
[0057] Figure 1 This is a schematic diagram of a multi-target detection and recognition method for dashboards and signs based on edge computing, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the corresponding edge computing-based multi-target detection and recognition system for dashboards and signs; such as... Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a multi-target monitoring and identification method for dashboards and signs based on edge computing, including:
[0058] Step 1: Collect image data containing dashboards and signs, construct a multi-target recognition dataset, and label the dashboards and signs in the dataset.
[0059] Step 2: Build a data monitoring algorithm to monitor whether one or more new images are transferred to a specified path such as a target folder.
[0060] Step 3: Build a YOLO multi-object detection deep neural network to detect dashboards and signs in new images, and segment the dashboard area and sign area.
[0061] Step 4: Build a multi-target recognition algorithm to recognize the dashboard readings in the dashboard area and the sign text in the sign area, and transmit the recognition results back to the cloud or output them in the interactive interface.
[0062] Step 5: Introduce an anomaly detection algorithm to analyze whether the dashboard readings are abnormal, and issue a warning message if any are found.
[0063] Step 6: Deploy the optimized model to edge devices to achieve real-time monitoring and identification of multiple targets and data feedback to the cloud.
[0064] The present invention deploys the deep neural network model to an edge device, which is used to transmit dashboard readings and sign text back to the cloud, or to output dashboard readings and sign text in an interactive interface.
[0065] In step 1, image data containing dashboards and corresponding facility signs are collected. The dashboard area, dashboard scale values, dashboard pointer, dashboard pointer reading, dashboard type, and sign area are labeled to obtain the class labels of each target class, forming a multi-target recognition dataset. The multi-target recognition dataset is then divided into a training dataset and a test dataset.
[0066] In step 2, the established data monitoring algorithm will be used to monitor whether single or multiple new image data are received along a specified path. Step 2 also includes the following steps:
[0067] Step 2.1: Define a specified path for receiving new images, such as a specified target folder; define a new mask to represent image creation events in the specified path.
[0068] Step 2.2: Create a monitoring list, add the specified paths defined in Step 2.1 to the monitoring list, and apply the mask defined in Step 2.1; create a monitor to manage the paths in the monitoring list.
[0069] Step 2.3: Create an event handler to process and distribute file system events. When the monitor in Step 2.2 detects an image creation event in the specified path, i.e., one or more new image data are created in the specified path, it reads these images, starts the multi-object detection model defined in subsequent steps, identifies dashboard readings and sign text in the images, and then deletes the identified images; conversely, if no image creation event occurs in the specified path, the multi-object detection model defined in subsequent steps is terminated.
[0070] In step 3, the constructed YOLO multi-object detection deep neural network is trained on the training dataset from step 1, and then detects and segments the dashboard and signage regions in the image. Step 3 also includes the following steps:
[0071] Step 3.1: The YOLO multi-object detection deep neural network uses a convolutional neural network to extract features from the input image of the dashboard and signage. These features include information such as edges, texture, and color, helping the model detect different objects in the image. YOLO (You Only Look Once) is an object detection algorithm.
[0072] Step 3.2: The YOLO multi-object detection deep neural network introduces a feature pyramid network to process feature maps in a multi-scale manner. This enables the model to detect both large-scale and small-scale targets simultaneously, thereby improving the accuracy of multi-object detection. The Feature Pyramid Network (FPN) passes high-level semantic features from top to bottom, which enhances the detection of small targets.
[0073] Step 3.3: The YOLO multi-object detection deep neural network uses predefined anchor boxes to predict the position and size of targets. By adjusting the size and scale of the anchor boxes, the YOLO multi-object detection deep neural network can better adapt to the shape and size of different target categories.
[0074] Step 3.4: After detecting a target, the YOLO multi-object detection deep neural network classifies each anchor box to determine the target's category. Simultaneously, to reduce duplicate detections, the YOLO multi-object detection deep neural network uses non-maximum suppression to retain the multi-object detection results with the highest confidence and remove overlapping bounding boxes.
[0075] Step 3.5: The YOLO multi-object detection deep neural network segments the dashboard area and the signage area based on the bounding boxes in Step 3.4. The segmented dashboard area and signage area are not only the detection results of the YOLO multi-object detection deep neural network in Step 3, but also the input images for the multi-object recognition algorithm in Step 4.
[0076] In step 4, to prevent errors in recognizing dashboard readings from angles other than the orthogonal view, the following steps are also included:
[0077] Step 4.1: Perform ellipse fitting on the dashboard area extracted in Step 3. Use the ellipse fitting function in OpenCV to generate the center position of the ellipse (i.e., the coordinates of the center point), the lengths of the major and minor axes, and the rotation angle of the ellipse, determining the shape and position of the dial area within the dashboard. After obtaining the ellipse fitting result, perform a spatial transformation on the result to correct the dashboard area from the original viewing angle to the normal viewing angle. A normal viewing angle means that the surface of the dashboard is perpendicular to the observer's line of sight, making the readings more accurate and clear. To achieve this, firstly, calculate the corresponding rotation matrix (containing information about the rotation angle and rotation center) based on the ellipse's rotation angle and center position. Use this rotation matrix to rotate the ellipse to a standard normal viewing position. Then, apply this rotation matrix to perform a spatial transformation on the original dashboard image, adjusting the tilted or distorted dashboard area to the normal viewing angle, thus improving the accuracy and reliability of dashboard reading recognition.
[0078] Step 4.2: Read the nearest numerical scale surrounding the instrument panel pointer using Optical Character Recognition (OCR) technology, and calculate the instrument panel reading using linear interpolation and angle ratio.
[0079] In step 4, to prevent errors in recognizing the text on the signage due to different lighting conditions, the following steps are also included:
[0080] Step 4.3: Use the grayscale image conversion tool in OpenCV to convert the sign area image extracted in Step 3 into a grayscale image, making the difference between the text on the sign and the background more obvious and simplifying the computational complexity of subsequent steps.
[0081] Step 4.4: Define the threshold for binarization. Based on this threshold, the grayscale image obtained in Step 4.4 is binarized to remove background noise, making the text on the sign clearer. This ensures that the multi-target recognition algorithm performs consistently well in the sign area under different lighting conditions, thus improving the robustness of the multi-target recognition algorithm.
[0082] Step 4.5: Configure the parameters of the Tesseract-OCR engine (Tesseract is an open-source optical character recognition (OCR) engine), define the optical character recognition mode, define the page segmentation mode, define the language of optical character recognition, specify the allowed character set according to relevant industry standards, use the optical character recognition technology in the Tesseract-OCR engine to process the binarized image in Step 4.5, segment the recognition result by spaces, and extract the sign text.
[0083] In step 4, the dashboard readings and sign text obtained in steps 4.2 and 4.5 are used as recognition results and fed back to the cloud in real time through a TCP / IP-based communication module, or directly output in the interactive interface.
[0084] In step 5, firstly, based on existing industry experience and expert knowledge, a reasonable range for instrument panel readings is determined for different situations. Then, an anomaly detection algorithm is introduced based on this range. When the algorithm reads an instrument panel reading and analyzes it to determine that the value is not within the reasonable range, the algorithm will issue a warning message so that relevant personnel can check, thus meeting the anomaly detection requirements in multi-target monitoring and identification scenarios involving instrument panels and signs.
[0085] In step 6, the multi-target detection and recognition algorithm and system from the previous steps are first converted from PyTorch files to ONNX files, and then from ONNX files to TensorRT files, so that the algorithm and system can be deployed to edge devices for real-time operation. Then, the TensorRT model parameters are half-precision compressed, i.e., compressed from floating-point 32 (FP32) to floating-point 16 (FP16), to reduce power consumption, reduce memory usage, save bandwidth, and improve computing speed. This enables the system to process data accurately and efficiently in real time, saving significant computing resources and storage space.
[0086] Figure 3This is a schematic flowchart of a multi-target detection and recognition method for dashboards and signs provided in another embodiment of the present invention; as shown below. Figure 3 As shown in the figure, the multi-target detection and recognition method for dashboards and signs provided by this embodiment of the invention includes:
[0087] Acquire an image to be detected, wherein the image to be detected is image data containing a dashboard and / or signage;
[0088] The image to be detected is input into a pre-trained multi-object detection model. The multi-object detection model detects dashboards and / or signs in the image to be detected and segments dashboard area images and / or sign area images. The multi-object detection model uses a YOLO-based deep neural network.
[0089] Recognize dashboard readings in the dashboard area image and / or signage text in the signage area image, and output the recognition results.
[0090] In some embodiments, the method further includes: a step of training a multi-object detection model, specifically including:
[0091] A multi-target recognition dataset is obtained, which includes multiple sample images containing dashboards and signs, as well as annotation information for the sample images. The annotation information includes annotations for dashboard areas, dashboard scale values, dashboard pointers, dashboard pointer readings, dashboard types, and sign areas. The multi-target recognition dataset is divided into a training dataset and a test dataset.
[0092] The training dataset is input into a pre-defined multi-object detection model for training, resulting in a trained multi-object detection model.
[0093] In some embodiments, the multi-object detection model includes a backbone network, a neck network, and a head network. The backbone network uses a convolutional neural network to extract features of the dashboard and signage from the input image, outputting multi-scale feature maps. These features include information such as edges, textures, and colors to help the model detect different objects in the image. The neck network uses a Feature Pyramid Network (FPN) to fuse the multi-scale features extracted by the backbone network to generate an enhanced feature map. The head network makes predictions based on the fused enhanced feature maps, outputting the bounding box coordinates, confidence scores, and class probabilities of the targets, and segmenting the dashboard region and / or signage region according to the bounding boxes of the targets.
[0094] In some embodiments, after the steps of outputting the bounding box coordinates, confidence scores, and class probabilities of the target, the method further includes: using a non-maximum suppression algorithm to retain the bounding box with the highest confidence score and removing overlapping bounding boxes.
[0095] In some embodiments, the step of identifying dashboard readings in the dashboard area image and / or signage text in the signage area image and outputting the identification result includes:
[0096] Ellipse fitting is performed on the extracted dashboard area image. The ellipse fitting function in OpenCV is used to generate the center position of the ellipse, the length of the major axis and the length of the minor axis, as well as the rotation angle of the ellipse.
[0097] After obtaining the ellipse fitting result, a spatial transformation is performed on the result to correct the dashboard area image (extracted dashboard area image) from the original viewpoint to the dashboard area image from the front viewpoint.
[0098] The instrument panel reading is calculated by reading the nearest numerical scale surrounding the instrument panel pointer in the instrument panel area image from a normal viewing angle using optical character recognition (OCR) and linear interpolation and angle ratio.
[0099] In some embodiments, the step of identifying dashboard readings in the dashboard area image and / or signage text in the signage area image and outputting the identification result includes:
[0100] The extracted sign area image is converted to grayscale using the grayscale image conversion tool in OpenCV to generate a grayscale image;
[0101] Define a threshold for binarization processing, and perform binarization processing on the grayscale image based on this threshold to remove background noise, making the text part of the sign clearer, so that the multi-target recognition algorithm has more consistent performance in the sign area under different lighting conditions, and improves the robustness of the multi-target recognition algorithm.
[0102] The binarized image is processed using Optical Character Recognition (OCR) to extract the text from the sign.
[0103] In some embodiments, the output recognition result is fed back to the cloud server in real time through a communication module based on the TCP / IP protocol, or is directly output in the interactive interface.
[0104] In some embodiments, the step of acquiring the image to be detected further includes a step of monitoring whether one or more images to be detected are transmitted to a specified path before the step of acquiring the image to be detected:
[0105] Define a specified path for receiving the image to be detected;
[0106] Define a mask to represent the image creation event in the specified path;
[0107] Create a monitoring list, add the specified path to the monitoring list, and apply the mask;
[0108] Create a monitor to manage the specified paths in the monitoring list;
[0109] Create event handlers to process and dispatch file system events;
[0110] When the monitor detects an image creation event in a specified path, that is, when one or more new image data are created in the specified path, it executes the step of acquiring the image to be detected, starts the multi-target detection model defined in the subsequent steps, identifies the dashboard readings and sign text in the image, and then deletes the identified image; otherwise, if there is no image creation event in the specified path, it terminates the execution of the multi-target detection model defined in the subsequent steps.
[0111] In some embodiments, the step of identifying dashboard readings in the dashboard area image and / or signage text in the signage area image, and outputting the identification result, includes the following:
[0112] The system compares the instrument panel readings in the identified instrument panel area image with the preset instrument panel reading thresholds. If not, a warning message is issued.
[0113] Based on the same inventive concept, embodiments of the present invention also provide a dashboard and signboard multi-target detection and recognition system, configured to implement any of the methods described in the above embodiments of the dashboard and signboard multi-target detection and recognition method, the system comprising:
[0114] The acquisition module is used to acquire the image to be detected;
[0115] The model processing module is used to input the image to be detected into a pre-trained multi-object detection model. The multi-object detection model detects dashboards and / or signs in the image to be detected and segments the dashboard area image and / or sign area image. The multi-object detection model adopts a YOLO-based deep neural network.
[0116] The recognition module is used to recognize dashboard readings in the dashboard area image and / or signage text in the signage area image, and output the recognition results.
[0117] In some embodiments, the system further includes an edge device, wherein the multi-target detection model is compressed into a half-precision model, converted into a TensorRT model, and then deployed on the edge device. Deploying the optimized model to the edge device enables real-time monitoring and identification of multiple targets and cloud data feedback.
[0118] The embodiments described above are merely preferred embodiments for fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.
Claims
1. A dashboard and signboard multi-target detection and recognition method, characterized in that, The method comprises the following steps: acquiring an image to be detected; inputting the image to be detected into a pre-trained multi-target detection model, the multi-target detection model detecting a dashboard and / or a signboard in the image to be detected, and segmenting out a dashboard region image and / or a signboard region image, the multi-target detection model using a deep neural network based on YOLO; recognizing a dashboard reading in the dashboard region image and / or a signboard text in the signboard region image, and outputting a recognition result.
2. The method of claim 1, wherein, The method further comprises the following steps of training the multi-target detection model, specifically comprising: acquiring a multi-target recognition data set, the multi-target recognition data set comprising a plurality of sample images containing a dashboard and a signboard and annotation information of the sample images, the annotation information comprising annotation of a dashboard region, a dashboard scale value, a dashboard pointer, a dashboard pointer reading, a dashboard type, and a signboard region, the multi-target recognition data set being divided into a training data set and a test data set; inputting the training data set into a preset multi-target detection model for training, to obtain a trained multi-target detection model. The multi-target detection model comprises a backbone network, a neck network, and a head network; 3. The method of claim 1, wherein, the backbone network uses a convolutional neural network to extract features of the dashboard and the signboard from the input image, and outputs multi-scale feature maps; the neck network uses a feature pyramid network to fuse the multi-scale features extracted by the backbone network, and generates enhanced feature maps; the head network performs prediction based on the fused enhanced feature maps, and outputs a bounding box coordinate, a confidence, and a class probability of a target, and segments out a dashboard region and / or a signboard region according to the bounding box of the target. After the step of outputting the bounding box coordinate, the confidence, and the class probability of the target, the method further comprises the step of using a non-maximum suppression algorithm to retain the bounding box with the highest confidence and remove overlapping bounding boxes.
4. The method of claim 3, wherein, The step of recognizing the dashboard reading in the dashboard region image and / or the signboard text in the signboard region image, and outputting a recognition result comprises the following steps:
5. The method of claim 3, wherein, performing ellipse fitting on the dashboard region image to generate a center position of an ellipse, a length of a major axis of the ellipse, a length of a minor axis of the ellipse, and a rotation angle of the ellipse; after obtaining the result of the ellipse fitting, performing spatial transformation on the result of the ellipse fitting to correct the dashboard region image to a dashboard region image at a front view angle; reading the nearest numerical scale surrounding the dashboard pointer in the dashboard region image at the front view angle by using an optical character recognition method, and calculating the dashboard reading by using linear interpolation and angle proportion. The step of recognizing the dashboard reading in the dashboard region image and / or the signboard text in the signboard region image, and outputting a recognition result comprises the following steps:
6. The method of claim 3, wherein, performing grayscale image conversion on the extracted signboard region image to generate a grayscale image; defining a threshold value for binary processing, and performing binary processing on the grayscale image based on the threshold value to remove background noise; processing the binary image by using an optical character recognition method to extract the signboard text. Before the step of acquiring the image to be detected, the method further comprises the step of monitoring whether a single or multiple images to be detected are transmitted to a specified path:
7. The method of claim 1, wherein, defining a specified path for receiving the image to be detected; Defining a mask representing an image creation event in a specified path; Creating a monitoring list, adding the specified path to the monitoring list, and applying the mask; Creating a monitor for managing the specified path in the monitoring list; Creating an event handler for processing and distributing file system events; When the monitor detects an image creation event in the specified path, performing the step of obtaining the image to be detected; When the monitor does not detect an image creation event in the specified path, not performing the step of obtaining the image to be detected.
8. The method of claim 1, wherein, The step of outputting the recognition result after identifying the instrument panel reading in the instrument panel area image and / or the signboard text in the signboard area image includes: Comparing whether the instrument panel reading in the instrument panel area image is within a preset instrument panel reading threshold range, and if not, issuing a warning prompt information.
9. A dashboard and signboard multi-target detection and recognition system, characterized in that, The system is configured to implement the method of any one of claims 1-8, and the system comprises: An acquisition module for acquiring an image to be detected; A model processing module for inputting the image to be detected into a pre-trained multi-target detection model, the multi-target detection model detecting an instrument panel and / or a signboard in the image to be detected, and segmenting out an instrument panel area image and / or a signboard area image, the multi-target detection model using a deep neural network based on YOLO; An identification module for identifying an instrument panel reading in the instrument panel area image and / or a signboard text in the signboard area image, and outputting a recognition result.
10. The system of claim 9, wherein, The system further comprises an edge device, and the multi-target detection model is compressed into a half-precision model and converted into a TensorRT model, and then deployed in the edge device.