Panel detection and identification method, device and equipment and computer readable storage medium

Abstract

The invention discloses a panel detection and recognition method, device and equipment and a computer readable storage medium, and relates to the technical field of panel detection, optical character recognition and the like, the method combines panel detection and optical character recognition technologies, analysis and extraction of panel expression content information and service logic information are achieved, and the method has the advantages of being high in practicability and high in practicability. Structured processing of identification result data is realized by utilizing a hierarchical sorting algorithm, and meanwhile, definition and analysis of processing configuration items are introduced, so that display and analysis of identification contents and data analysis are facilitated, the identification effect is improved, an expected identification effect can be achieved especially when a page with complex business logic is identified, and the identification efficiency is improved. And panel detection applications are simpler and more convenient.

Description

Technical Field

[0001] This application relates to the technical fields of panel inspection and optical character recognition, and specifically to a panel inspection and recognition method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Panel detection and Optical Character Recognition (OCR) are two important directions in computer vision, each with unique application scenarios and technical implementations, while also being interconnected and complementary in some aspects. Panel detection aims to identify and locate specific panel areas in an image, such as dashboards, electronic screens, and billboards. Its applications include intelligent monitoring systems, traffic sign recognition in autonomous driving, and screen content analysis in smart home devices. The core objective of OCR is to extract readable text information from an image. With technological advancements, OCR has been widely applied in document scanning, license plate recognition, ticket processing, and intelligent typesetting. Its applications are not limited to static images but have also extended to text recognition in video streams.

[0003] However, there is no reliable method for processing OCR recognition results in the field of panel detection, especially for some complex device panels. Summary of the Invention

[0004] To address the limitations of existing panel inspection technologies, this application proposes a panel inspection and recognition method, apparatus, device, and computer-readable storage medium. By combining panel inspection with optical character recognition technology, it enables the analysis and extraction of panel content information and business logic information, thereby improving the recognition effect.

[0005] This application is achieved through the following technical solution:

[0006] A panel detection and recognition method, comprising:

[0007] Acquire an image of the device panel and identify the display screen and indicator lights of the device panel from the image;

[0008] The text information displayed on the display screen is identified using a pre-trained text detection and recognition model, and the status information displayed on the display screen is identified using image processing technology; wherein, the identification results of the text information and the status information both include the corresponding information obtained and its relative position information relative to the display screen;

[0009] A hierarchical sorting algorithm is used to convert the recognition results of the text information and the recognition results of the status information into one-dimensional arrays respectively, and then merge them to form a complete one-dimensional array.

[0010] The one-dimensional array formed by fusion and the business logic definition are used to process the configuration items. The one-dimensional array formed by fusion is mapped to the object list where the processing configuration items are located. Based on the processing configuration items, the recognition results are clustered according to the page logic.

[0011] In some embodiments, the method of recognizing text information displayed on the display screen using a pre-trained text detection and recognition model, and recognizing status information displayed on the display screen using image processing technology, includes:

[0012] The identified display screen image is preprocessed to generate input vector data.

[0013] The input vector data is fed into a text detection model to perform text detection and obtain the detection result; the detection result is a text detection box.

[0014] Based on the detection results, the display screen image is cropped to obtain an image list containing only text;

[0015] The image list is preprocessed to generate a vector data list;

[0016] The vector data list is input into the text recognition model for inference, and finally a list of text recognition results is obtained. The list of results and the detection results together constitute the recognition result of the text information.

[0017] The identified display screen image is converted to the HSV color space, and then the color information representing the state in the display screen is extracted by the OpenCV color extraction algorithm to obtain the state information recognition result, which includes the state information and its position information relative to the display screen.

[0018] In some implementations, the step of using a hierarchical sorting algorithm to convert the recognition results of the text information and the recognition results of the state information into one-dimensional arrays includes:

[0019] The recognition result is characterized by the corner coordinates of the recognition result;

[0020] All recognition results are layered according to the ordinate values ​​of the corner coordinates of the recognition results; and the recognition results of each layer are sorted according to the abscissa values ​​of the corner coordinates of the recognition results.

[0021] The sorted recognition results are concatenated into an ordered one-dimensional array.

[0022] In some implementations, the step of stratifying all recognition results according to the ordinate values ​​of the corner coordinates of the recognition results, and sorting the recognition results of each layer according to the abscissa values ​​of the corner coordinates of the recognition results, includes:

[0023] Sort all the recognition and detection results according to the Y direction to obtain set B;

[0024] Traverse the set B;

[0025] Layered sorting steps: Store data items in a pre-established set C, determine whether the absolute value of the coordinate difference between the current data item and the next data item in the Y direction exceeds the layering threshold set in the Y direction. The layering threshold is set according to the pixel interval in the Y direction of each layer of data. If it does not exceed the threshold, continue to repeat this step to add data items to the set C.

[0026] If the hierarchical threshold is exceeded, the set C is sorted in the X direction, and all sorted data items in the set C are stored in a pre-established set D, and then the set C is cleared.

[0027] Continue traversing set B, repeating the hierarchical sorting steps, until all data items in set B have been traversed.

[0028] In some embodiments, before fusing the recognition result of the text information with the recognition result of the state information, the method further includes:

[0029] When the recognition result of the text information or the recognition result of the status information does not match the actual situation, the recognition result of the text information and the recognition result of the status information are processed linearly.

[0030] In some embodiments, after fusing the recognition results of the text information and the recognition results of the status information, the method further includes:

[0031] Based on the recognition result generated after fusion, it is determined whether dynamic information exists; if so, dynamic information detection and recognition are performed. The process of dynamic information detection and recognition includes:

[0032] The dynamic information is removed from the recognition results by using keywords and then added to the cache list;

[0033] The specific business module to which the dynamic information belongs is determined by the anchor points of the static information retained in the recognition results.

[0034] In some implementations, the step of processing configuration items based on the fused one-dimensional array and business logic definitions, mapping the fused one-dimensional array to the object list containing the processing configuration items, and clustering the recognition results based on the processing configuration items according to page logic includes:

[0035] The processing configuration items are parsed into a list of objects, including static information configuration and module-based processing configuration.

[0036] All the recognition results are mapped to the object list where the processing configuration item is located, except for static information which does not need to be mapped.

[0037] Clustering is performed according to the modular processing configuration.

[0038] Secondly, this application proposes a panel detection and recognition device, comprising:

[0039] The first identification unit is used to acquire an image of the device panel and identify the display screen and indicator lights of the device panel from the image of the device panel;

[0040] The second recognition unit is used to recognize the text information displayed on the display screen using a pre-trained text detection and recognition model, and to recognize the status information displayed on the display screen using image processing technology; wherein, the recognition results of the text information and the status information both include the corresponding information obtained and its relative position information relative to the display screen;

[0041] The conversion and fusion unit is used to use a hierarchical sorting algorithm to convert the recognition results of the text information and the recognition results of the state information into one-dimensional arrays respectively and fuse them into a complete one-dimensional array.

[0042] And a parsing unit, configured to process configuration items according to the fused one-dimensional array and business logic definition, map the fused one-dimensional array to the object list where the processing configuration items are located, and perform clustering processing on the recognition results based on the processing configuration items and the page logic situation.

[0043] Thirdly, this application proposes an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the above-described embodiments of the panel detection and recognition method.

[0044] Fourthly, this application proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described embodiments of the panel detection and recognition method.

[0045] This application proposes a panel detection and recognition method that combines panel detection with optical character recognition technology to analyze and extract the content information and business logic information expressed on the panel. It also uses a hierarchical sorting algorithm to achieve structured processing of the recognition result data. At the same time, it introduces the definition and parsing of processing configuration items to facilitate the display, analysis and data analysis of the recognized content, thereby improving the recognition effect. In particular, it can achieve the expected recognition effect for pages with complex business logic, making panel detection applications simpler and more convenient.

[0046] Accordingly, the panel detection and recognition device, computer equipment, and computer-readable storage medium proposed in this application also possess the same technical effects as described above. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of the embodiments of this application and form part of this application, do not constitute a limitation on the embodiments of this application. In the drawings:

[0048] Figure 1 This is a flowchart of the identification method proposed in the embodiments of this application;

[0049] Figure 2 This is a schematic diagram of the analog device panel according to an embodiment of this application;

[0050] Figure 3 for Figure 2 The diagram shows the indicator light recognition results of the simulated device panel.

[0051] Figure 4 for Figure 2 A schematic diagram of the display screen recognition results of the simulated device panel shown;

[0052] Figure 5 For the purposes of this application's embodiments Figure 4 The recognition results shown on the display screen;

[0053] Figure 6 The hierarchical sorting algorithm used in this application embodiment is for... Figure 5 The recognition results and processing results are shown below;

[0054] Figure 7 In this embodiment of the application, the corner coordinates of data items on the same layer are not on the same layer;

[0055] Figure 8 for Figure 5 The recognition results shown are the final recognition results after hierarchical sorting and fusion;

[0056] Figure 9 This is a schematic diagram for detecting dynamic information;

[0057] Figure 10 This refers to the definition and parsing results of the processing configuration items in this application embodiment;

[0058] Figure 11 This is a flowchart illustrating the OCR recognition process of an embodiment of this application.

[0059] Figure 12 This is a block diagram illustrating the principle of the identification device proposed in the embodiments of this application;

[0060] Figure 13 This is a schematic diagram of the identification system architecture proposed in an embodiment of this application;

[0061] Figure 14 This is a schematic diagram of the electronic device proposed in the embodiments of this application;

[0062] Figure 15 This is a schematic diagram of a computer-readable storage medium proposed in an embodiment of this application;

[0063] Figure reference numerals and corresponding component names:

[0064] 200 - Identification device, 201 - First identification unit, 202 - Second identification unit, 203 - Conversion and fusion unit, 204 - Parsing unit, 300 - Identification system, 301 - Input device, 302 - Output device, 303 - Processor A, 304 - Memory A, 400 - Electronic device, 410 - Memory B, 420 - Processor B, 411 - Computer program A, 500 - Computer-readable storage medium, 511 - Computer program B. Detailed Implementation

[0065] In the following, the terms “comprising” or “may include” as used in the various embodiments of this application indicate the presence of a function, operation, or element of the invention and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in the various embodiments of this application, the terms “comprising,” “having,” and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or adding one or more combinations of the foregoing.

[0066] In various embodiments of this application, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0067] The terms used in the various embodiments of this application (such as "first," "second," etc.) may modify various constituent elements in the various embodiments, but do not limit the corresponding constituent elements. For example, the above terms do not limit the order and / or importance of the elements. The above terms are only used for the purpose of distinguishing one element from other elements. For example, a first user device and a second user device refer to different user devices, although both are user devices. For example, without departing from the scope of the various embodiments of this application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0068] It should be noted that if a description is made of "connecting" one component to another, then the first component can be directly connected to the second component, and a third component can be "connected" between the first and second components. Conversely, when a component is "directly connected" to another component, it can be understood that there is no third component between the first and second components.

[0069] The terminology used in the various embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. The terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.

[0071] This application proposes a panel detection and recognition method that combines optical character recognition and panel detection technology to achieve universal device panel detection. This method can be applied to all panel detection application scenarios, improves the accuracy of recognition, and can even recognize pages with complex business logic, achieving the expected recognition effect.

[0072] like Figure 1 As shown, the panel detection and recognition method proposed in this application includes the following steps:

[0073] Step 1: Acquire an image of the device panel and identify the main elements of the device panel from the image; the main elements identified include, but are not limited to, the display screen, indicator lights, etc.

[0074] Step 2: Use a pre-trained text detection and recognition model to identify the text information displayed on the screen, and use image processing technology to identify the status information displayed on the screen;

[0075] Step 3: Use a hierarchical sorting algorithm to convert the text information recognition results and the state information recognition results into one-dimensional arrays respectively, and then merge them to form a complete one-dimensional array;

[0076] Step 4: Based on the fused one-dimensional array and the business logic definition, process the configuration items and map the fused one-dimensional array to the object list where the processing configuration items are located. Then, based on the processing configuration items, perform clustering processing on the recognition results according to the page logic.

[0077] Furthermore, the specific implementation process of step 1 in this application embodiment is as follows:

[0078] For equipment panel image acquisition, this application embodiment uses a camera to realize real-time acquisition of equipment panel images. The requirements for the acquisition equipment environment are as follows: (1) Camera configuration: First, a high-resolution industrial camera is required. On the one hand, the camera needs to work continuously to ensure the practicality and reliability of the camera. On the other hand, text detection requires a clear picture, especially to capture clear text information to improve the accuracy of OCR recognition. The captured picture should also have distinct colors to facilitate signal light recognition. Secondly, it needs to support the configuration of common camera parameters, such as camera exposure, brightness, contrast, focal length, etc., to improve the quality of the captured picture through parameter configuration. (2) Image acquisition configuration: Image acquisition can use video stream, shooting, etc., and can be selected according to specific business needs. For high real-time requirements, video stream is used, and for low real-time requirements, shooting is used. In addition, for low real-time requirements, the CPU can be used, and for high real-time requirements, the GPU is introduced. (3) Illumination effect: When the image acquisition frequency is low, use dynamic parameter configuration, that is, use two different sets of exposure parameters (switching between low exposure parameters and high exposure parameters) at different time periods to realize the acquisition of the device panel; when the image acquisition frequency is high, a set of low exposure static parameters (low exposure parameters) can be used to realize the acquisition of the device panel. Under low exposure conditions, the indicator lights and display screen of the device panel can be clearly captured. The disadvantage is that only the display screen and the indicator lights of the device panel can be identified, and other information of the device panel cannot be extracted. The specific choice should be made according to the actual application scenario; in addition, the shooting light should be as uniform as possible to avoid reflection and shadow; (4) Installation conditions: maintain a fixed state, requiring that the camera and the device be relatively fixed, including the distance between the camera and the device, the camera rotation angle, etc.; the working environment temperature, humidity and other working environments of the camera and the device should be matched so as not to affect the normal maintenance and use of the device.

[0079] Identify device indicator lights from the acquired device panel images. Device indicator lights are signal lights on the device panel used to determine certain states, including on / off status, connection status, running status, self-test status, and all other extractable status information. Device indicator lights can be distinguished by different colors and brightness levels. Their relative positions must also be identified to determine the indicator lights representing different meanings. Figure 2 The indicator lights on the simulated device panel are shown below. Figure 3As shown. The identification process of the device indicator lights is as follows:

[0080] Image conversion: Converting a raw RGB image to HSV color space involves transforming the superimposed table of the red, green, and blue channels in the RGB image into a representation of hue (H), saturation (S), and lightness (V) in the HSV color space. First, the values ​​of the three RGB channels need to be normalized to the range [0,1], i.e., the color channel values ​​are divided by 255. The conversion formula is as follows:

[0081]

[0082] in These represent the red, green, and blue coordinates, respectively, and their values ​​are real numbers between 0 and 1; , indicates taking the maximum value; , indicates taking the minimum value; These represent the values ​​of hue (H), saturation (S), and lightness (V), respectively.

[0083] Color threshold filtering: Based on the color of the target indicator light, determine its threshold range in the HSV space. For example, the HSV range of a red indicator light might be [340°, 0.6, 0.6] to [20°, 1, 1], meaning the H value is between 340° and 20°, the S value is between 0.6 and 1, and the V value is between 0.6 and 1. The relevant color areas of the panel are filtered out using color thresholding and then binarized to generate a binarized mask image.

[0084] Morphological processing: Erosion and dilation operations are performed on the binarized mask image to remove noise and small interference areas, and to find the contours of connected regions.

[0085] Indicator light recognition: Based on the area, roundness, and other features of the connected region outline, regions that match the shape of the indicator light are selected, the selected regions are located, and the status of the indicator light is determined.

[0086] Simultaneously, the device display screen is identified from the acquired device panel image. Display screen recognition aims to provide the minimum image recognition range for OCR content recognition. Generally, directly using the display screen for recognition improves accuracy and increases video frame processing speed. Display screen recognition only occurs before video processing begins. After obtaining the display screen's position relative to the device panel, the relative position information is cached. During processing, the display screen position is directly retrieved from the cache, and the processing flow continues after the original device panel image has been cropped. Figure 2 The display screen after the analog device panel is cropped is shown below. Figure 4 As shown. The device's display screen recognition process is as follows:

[0087] Grayscale processing of device panel images: A topology-based contour tracking algorithm is used to first process the device panel image into a grayscale image;

[0088] Gaussian blur processing: The obtained grayscale image is subjected to Gaussian blur processing;

[0089] Edge detection: Perform edge detection on the image after Gaussian blurring, specifically using the Canny edge detection method;

[0090] Closed Edges: Closes the edges detected by edge detection;

[0091] Contour extraction: Find all contours from closed edges;

[0092] Select the outlines that meet the criteria and crop the device panel image to obtain the device display screen.

[0093] This application embodiment obtains all the contour information of the device panel image through a contour extraction process, then filters out the contours that match the device display screen by contour size, and finally crops the device panel image by contour lines to obtain the device display screen.

[0094] Furthermore, in step 2 of the embodiments of this application, the content displayed on the device display screen mainly includes text information, status information, and dynamic information.

[0095] The information displayed on the device screen is mostly text. This text is displayed in three states: static text, changing text, and dynamic text. Static text includes titles, headers, and other information that remains unchanged on the screen. Changing text is text that is constantly updated on the screen, such as… Figure 4 The working status of module D on the device display screen can be either faulty or normal, but it will not be undisplayed. Dynamic text refers to text information that may or may not be displayed on the device display screen. Static text can be left unprocessed during recognition; a data dictionary can be directly established. For changing and dynamic text, OCR recognition must be used. During the recognition process, it is necessary to distinguish between them. OCR recognition mainly uses a preset text detection and recognition model to perform model inference on the device display screen image. The OCR recognition process is as follows: Figure 11As shown, the text detection and recognition model uses the PaddleInference library for inference. First, the text detection model and text recognition model are loaded using the PaddleInference library. Then, image preprocessing operations such as image resizing, normalization, and channel conversion are performed on the input device display image to obtain vector data for the text detection model. This vector data is then input into the text detection model for text detection to obtain the detection results. Based on the text detection results, i.e., the text detection boxes, the device display image is cropped to obtain a list of images containing only text. These images then undergo further preprocessing processes such as image resizing, normalization, and channel conversion to obtain a list of vector data for text recognition. This list of vector data is then input into the text recognition model for inference, ultimately yielding a list of text recognition results.

[0096] Status information refers to information displayed on a device's screen, excluding text information, that can be used to describe certain states. For example, screen brightness can be used as a basis for determining whether the device is on or off when it lacks a power switch indicator. Different colors can represent different business-related modules, such as... Figure 4 The device display shows the enabled status of each module in the lower right corner. Status information detection and recognition utilizes the HSV color space and is implemented through an image processing library. First, the elements representing status information on the display are analyzed to determine whether different states are distinguished by different colors. Then, color detection is implemented using a library provided by OpenCV.

[0097] The specific steps are as follows: (1) Convert the image to HSV format, (2) Set the color range, (3) Binarize the image, (4) Contour recognition, (5) Calculate the detection box. The detection box indicates that the color to be detected exists at the relevant pixel position in the image. Then, the specific meaning of the corresponding business is given. The detection box of the detection result is similar to the detection box of text detection, indicating the specific pixel position of the detection information in the image.

[0098] Dynamic information refers to pop-ups that appear on the device's display screen during operation, providing summary information such as device self-test results. These pop-ups may disappear after a few seconds, requiring a camera to capture and recognize this state. Dynamic information detection typically includes text or color information. If it is text, OCR is used for detection and recognition, followed by text matching. If it is color, the HSV color space is used for detection. Dynamic information detection is independent of the above detection process.

[0099] Furthermore, in step 3 of this embodiment, the OCR recognition result is a set of OCR data items (text detection result + confidence score + detection box). Ignoring the device display screen itself, the distribution of the OCR recognition result is as follows: Figure 5As shown, it is a two-dimensional planar distribution in the horizontal (X) and vertical (Y) directions. The Y-axis represents the height of the image to be detected (i.e., the image on the device display screen), and the X-axis represents the width of the image to be detected. The figure shows that the OCR recognition results are distributed in a two-dimensional plane on the device display screen. Assuming these recognition results are represented by their corner points, for example... Figure 5 The point P(x, y) in the upper left corner represents these recognition results. Using the positional information of these results in the image, the recognition results can be converted into a one-dimensional array arranged according to a certain pattern. First, the data can be layered in the Y direction, that is, layered according to the y coordinate of point P (specifically, within a certain height range of the image, there are multiple recognition results; after sorting the detection results along the Y axis, traversing the set, the data in different layers will show obvious gradient changes; for example, the y-value change of the detection results in the same layer is generally within 10 pixels, but when traversing to a new element, the y-value change of the pixel exceeds 10 pixels, indicating that the next layer has been reached; that is, data layering is the differentiation of detection results according to the pixel height of the image). Then, the data is sorted in the X direction, that is, the layered data is sorted according to the x coordinate of point P, resulting in an ordered one-dimensional array in the X and Y directions, as shown below. Figure 6 The results shown are sorted by layering in the Y direction and sorting in the X direction and then concatenated into an ordered one-dimensional array.

[0100] Specifically, the algorithm for hierarchically sorting the recognition results is as follows:

[0101] The data items in the OCR recognition result set A are sorted according to the Y direction to obtain set B. This step aims to make the entire detection result ordered by sorting by the y coordinate, so that the set can be traversed sequentially and the data from the same layer can be determined according to the set threshold.

[0102] Traverse set B;

[0103] Hierarchical sorting steps: Store data items in a pre-established set C, and determine whether the absolute value of the difference between the y-coordinate of the current data item and the next data item (for example, calculate the absolute value of the difference using the y-coordinate of the upper left corner of the detection box) exceeds the hierarchical threshold set in the Y direction. This hierarchical threshold is set based on experience. If it does not exceed the threshold, continue to add data items to set C in a loop.

[0104] If the hierarchical threshold is exceeded, sort set C in the X direction, store the sorted data items of set C into a pre-established set D, and then clear set C.

[0105] Continue iterating through set B, repeating the hierarchical sorting steps described above, until all data items in set B have been traversed.

[0106] Furthermore, when performing stratification, the corner points (e.g., corner points) between data items in each layer of the dataset are not fixed values, but fluctuate up and down. Therefore, the stratification threshold in the stratification sorting algorithm of the recognition result dataset must be greater than the maximum fluctuation range in order to distinguish the layers of data and obtain a useful one-dimensional array.

[0107] The status information recognition results are similar to the text information recognition results, both including the recognition result information and the relative position information relative to the display screen. Therefore, the two-dimensional data results are converted into a one-dimensional array according to the same Y-direction sealing layer X-direction sorting algorithm.

[0108] If the text information recognition results and the state information recognition results are directly fused, there may be issues such as... Figure 7 The problem is that the y-coordinates of points P and Q at the top left corner of the detection box are not on the same layer, but the detection boxes represented by points P and Q should be on the same layer. Directly using the y-coordinate for layering might result in them not being on the same layer. To address this, this embodiment introduces linear processing for points P1 and Q1 (i.e., the coordinates of the bottom left corner of the detection box), calculating the median y-coordinates of P and P1, and Q and Q1, thereby classifying the detection results represented by P and Q into the same layer. The coordinate transformation formula is as follows:

[0109]

[0110]

[0111] in, , These are the transformed coordinates of points P and Q, respectively. Let P and P1 be the y-coordinates of points P and P1, respectively. The y-coordinates of points Q and Q1 are respectively. Let P and Q be the x-coordinates of points P and Q, respectively.

[0112] At this point, the coordinates of the top left corner of all detection boxes need to be transformed to the point where the center line is located, and then the layers can be sorted.

[0113] The text information recognition results, converted into a one-dimensional array, are fused with the state information recognition results to form a complete one-dimensional array, and the meaning of each index in the array is clearly known. Figure 8 The results of hierarchical sorting and fusion of the identification results are shown.

[0114] If the device display includes dynamic text or involves dynamic information, it can be determined through OCR recognition results. Figure 8 Taking the recognition results shown as an example, the OCR detection results for static and dynamic text are 20 items. If dynamic text appears, or the number of pop-up detections is not equal to 20 items, the dynamic information will be processed separately.

[0115] Dynamic information detection and recognition requires analysis in conjunction with the specific business logic of the display screen. For dynamically displayed information (i.e., text or indicator lights with a fixed position on the display screen that may or may not be displayed), text needs to be first removed from the recognition results using keywords to ensure the correct logical order of the statically displayed information detected by OCR. Then, the specific business module to which the dynamic information belongs is determined by the anchor points of the static information. For dynamically displayed indicator lights, the determination is directly made by using the anchor points of the static information. Figure 4 Taking the device display screen shown as an example, if Figure 9 The diagram illustrating the detection of dynamic text shows that module A includes the dynamically displayed text "Connection Status: TRUE". This text will not be displayed if the device is not connected to the network; after a network cable is connected, the connection status will indicate whether the connection was successful. The addition of dynamically displayed information to the OCR recognition results will change all previous indexes. To maintain the hierarchical order of the previously statically displayed information, the dynamic information needs to be removed first. For example, using the keyword "Connection Status", the dynamic information items in the OCR recognition result list are first removed and added to the cache list. Then, the display of dynamic information in the cache list is processed. Since modules A, B, and C have the same structure, to accurately determine whether "Connection Status: TRUE" belongs to module A, B, or C, static information anchors are needed to determine which module the dynamic information belongs to. For the current panel, the top-left corner of the text detection box of module A can be used as the anchor point. Given a pixel range, it is determined whether "Connection Status: TRUE" is within that pixel range. If it is, it indicates that the dynamically displayed information belongs to "module A".

[0116] Furthermore, while the above steps can achieve panel content detection and recognition, and generate structured business data, the post-processing method is not very user-friendly. Therefore, it requires creating objects one by one through indexes to retrieve and assign values. Thus, step 4 of this embodiment introduces configuration item definition and parsing processing. Based on the configuration items, the recognition results can be clustered according to the page's business logic, increasing processing flexibility and simplifying the processing flow. The specific process is as follows:

[0117] Process configuration item definitions: such as Figure 10 As shown, the processing configuration items combine the recognition results with business logic and include configurations such as static information processing configuration and modular processing configuration. The processing configuration items define the processing logic of the program.

[0118] Configuration item parsing and processing: After obtaining the configuration items, they are parsed into a list of objects. First, all the recognition results are mapped to the list of objects where the configuration items are located. Static information does not need to be mapped. Then, the configuration is processed according to modules, that is, the business modules A, B, C, D, and E are categorized and processed by numerical identifiers. At this point, the data has a preliminary logic and can be used for display and data analysis.

[0119] The data can be further populated into specific objects. Figure 4 In the device display shown, A, B, and C are grouped into one category, and D and E into another. Therefore, two for loops can be used to complete the object assignment, avoiding the need to retrieve the index and assign values ​​one by one. This approach can be adapted to different data processing scenarios.

[0120] Furthermore, in this embodiment, the PaddleOCR deep learning model is selected as the basic model framework for panel detection. However, this model is a general-purpose model, applicable to all optical character recognition scenarios, but its recognition performance may be poor for specific scenarios. For example, in the LCD screen recognition applied in this embodiment, the text detection and recognition performance of the device screen using the PaddleOCR deep learning model does not achieve the expected results. Therefore, this embodiment enhances the training of the PaddleOCR deep learning model to serve as the final text detection and recognition model. The specific training process is as follows:

[0121] Dataset Construction: Images of the device panels to be identified are acquired using a camera. Several real images of the devices are obtained using strategies such as different lighting, distances, and display content (e.g., about 50 images). The number of image samples for each strategy is evenly distributed. Then, the data samples are enhanced by adjusting contrast, saturation, and color temperature to augment images of different colors. Image samples of different sharpness are augmented by adding image noise, ultimately generating more device panel images (e.g., about 100 images).

[0122] Use a semi-automatic annotation tool to label the data. Divide the labeled dataset into a text detection dataset and a text recognition dataset. Each dataset is further divided into a training set, a test set, and a validation set. When using the tool to divide the dataset, set the ratio of the training set, test set, and validation set to 6:2:2. Once the above steps are completed, the dataset creation is finished.

[0123] Selection of training model: In this embodiment, pre-trained weights suitable for the server are selected as pre-trained weights for the text detection model to improve the accuracy of text detection, and lightweight model weights are selected as pre-trained weights for the text recognition model.

[0124] Configuration and Training: Configure the dataset, the number of training iterations, and whether to enable the GPU according to the hardware environment. Train the model. If it fails to meet expectations, adjust the parameters and the dataset, and repeat the training until the expected results are achieved.

[0125] Based on the same technical concept described above, this application also proposes a panel detection and recognition device, such as... Figure 12 As shown, the identification device 200 includes:

[0126] The first identification unit 201 is used to acquire a device panel image and identify the main elements of the device panel from the image. The specific implementation is as described in step 1 above and will not be repeated here.

[0127] The second recognition unit 202 is used to recognize the text information displayed on the display screen using a pre-trained text detection and recognition model, and to recognize the status information displayed on the display screen using image processing technology. The specific implementation method is as described in step 2 above, and will not be repeated here.

[0128] The conversion and fusion unit 203 is used to convert the text information recognition results and the state information recognition results into one-dimensional arrays respectively using a hierarchical sorting algorithm and then fuse them into a complete one-dimensional array. The specific implementation method is as described in step 3 above, and will not be repeated here.

[0129] Furthermore, the parsing unit 204 is used to process configuration items based on the fused one-dimensional array and business logic definitions, map the fused one-dimensional array to the object list where the processing configuration items are located, and perform clustering processing on the recognition results based on the page logic according to the processing configuration items. The specific implementation method is as described in step 4 above, and will not be repeated here.

[0130] Furthermore, the identification device 200 in this application embodiment also includes:

[0131] The model training unit is used to train the PaddleOCR deep learning model as the basic model framework for panel detection to obtain the final text recognition and detection model.

[0132] Based on the same technical concept described above, this application also proposes a panel detection and recognition system, such as... Figure 13 As shown, the identification system 300 proposed in this application includes:

[0133] The system comprises an input device 301, an output device 302, a processor A303, and a memory A304; wherein the number of processors A303 and memory A304 can be one or more. Figure 13The following description uses a processor A303 and a memory A304 as an example. The input device 301, output device 302, processor A303, and memory A304 can be connected via a bus or other means. Figure 13 Taking the example of a connection between China and Israel via a bus.

[0134] Specifically, by calling the operation instructions stored in memory A304, processor A303 executes the following steps:

[0135] Acquire an image of the device panel and identify the main elements of the device panel from the image; wherein the main elements identified include, but are not limited to, the display screen, indicator lights, etc.

[0136] The text information displayed on the screen is identified using a pre-trained text detection and recognition model, and the status information displayed on the screen is identified using image processing technology.

[0137] A hierarchical sorting algorithm is used to convert the text information recognition results and the state information recognition results into one-dimensional arrays respectively and then merge them to form a complete one-dimensional array.

[0138] The fused one-dimensional array and business logic definition are used to process configuration items, and the fused one-dimensional array is mapped to the object list where the processing configuration items are located. Based on the processing configuration items, the recognition results are clustered according to the page logic.

[0139] Optionally, by calling the operation instructions stored in memory A304, processor A303 is also used to execute any of the embodiments in the corresponding examples of the above identification method.

[0140] Based on the same technical concept described above, this application also proposes an electronic device, such as... Figure 14 As shown, the electronic device 400 includes: a memory B410, a processor B420, and a computer program A411 stored in the memory B410 and executable on the processor B420. When the processor B420 executes the computer program A411, it performs the following steps:

[0141] Acquire an image of the device panel and identify the main elements of the device panel from the image; wherein the main elements identified include, but are not limited to, the display screen, indicator lights, etc.

[0142] The text information displayed on the screen is identified using a pre-trained text detection and recognition model, and the status information displayed on the screen is identified using image processing technology.

[0143] A hierarchical sorting algorithm is used to convert the text information recognition results and the state information recognition results into one-dimensional arrays respectively and then merge them to form a complete one-dimensional array.

[0144] The fused one-dimensional array and business logic definition are used to process configuration items, and the fused one-dimensional array is mapped to the object list where the processing configuration items are located. Based on the processing configuration items, the recognition results are clustered according to the page logic.

[0145] Optionally, when processor B420 executes computer program A411, it can implement any of the embodiments in the corresponding examples of the above-described identification method.

[0146] It should be noted that the electronic device proposed in this application embodiment is a device used to implement the above-mentioned identification method. Therefore, based on the above-mentioned identification method proposed in this application embodiment, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this application embodiment. Therefore, how the electronic device specifically implements the above-mentioned identification method will not be described in detail here. Any electronic device used by those skilled in the art to implement the above-mentioned identification method falls within the scope of protection of this application.

[0147] Based on the same technical concept described above, embodiments of this application also propose a computer-readable storage medium, such as... Figure 15 As shown, the computer-readable storage medium 500 stores a computer program B511, which, when executed by a processor, performs the following steps:

[0148] Acquire an image of the device panel and identify the main elements of the device panel from the image; wherein the main elements identified include, but are not limited to, the display screen, indicator lights, etc.

[0149] The text information displayed on the screen is identified using a pre-trained text detection and recognition model, and the status information displayed on the screen is identified using image processing technology.

[0150] A hierarchical sorting algorithm is used to convert the text information recognition results and the state information recognition results into one-dimensional arrays respectively and then merge them to form a complete one-dimensional array.

[0151] The fused one-dimensional array and business logic definition are used to process configuration items, and the fused one-dimensional array is mapped to the object list where the processing configuration items are located. Based on the processing configuration items, the recognition results are clustered according to the page logic.

[0152] Optionally, when the computer program B511 is executed by the processor, it can implement any of the embodiments corresponding to the above-described identification method.

[0153] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A panel detection and recognition method, characterized in that, include: Acquire an image of the device panel and identify the display screen and indicator lights of the device panel from the image; The text information displayed on the display screen is identified using a pre-trained text detection and recognition model, and the status information displayed on the display screen is identified using image processing technology; wherein, the identification results of the text information and the status information both include the corresponding information obtained and its relative position information relative to the display screen; A hierarchical sorting algorithm is used to convert the recognition results of the text information and the recognition results of the status information into one-dimensional arrays respectively, and then merge them to form a complete one-dimensional array. The one-dimensional array formed by fusion and the business logic definition are used to process the configuration items. The one-dimensional array formed by fusion is mapped to the object list where the processing configuration items are located. Based on the processing configuration items, the recognition results are clustered according to the page logic.

2. The panel detection and recognition method according to claim 1, characterized in that, The method of using a pre-trained text detection and recognition model to identify text information displayed on the screen, and using image processing technology to identify status information displayed on the screen, includes: The identified display screen image is preprocessed to generate input vector data. The input vector data is fed into a text detection model to perform text detection and obtain the detection result; the detection result is a text detection box. Based on the detection results, the display screen image is cropped to obtain an image list containing only text; The image list is preprocessed to generate a vector data list; The vector data list is input into the text recognition model for inference, and finally a list of text recognition results is obtained. The list of results and the detection results together constitute the recognition result of the text information. The identified display screen image is converted to the HSV color space and image processing is performed to obtain the identification result of the status information, including the status information and its position information relative to the display screen.

3. The panel detection and recognition method according to claim 1, characterized in that, The method of using a hierarchical sorting algorithm to convert the recognition results of the text information and the recognition results of the state information into one-dimensional arrays includes: The recognition result is characterized by the corner coordinates of the recognition result; All recognition results are layered according to the ordinate values ​​of the corner coordinates of the recognition results; and the recognition results of each layer are sorted according to the abscissa values ​​of the corner coordinates of the recognition results. The sorted recognition results are concatenated into an ordered one-dimensional array.

4. The panel detection and recognition method according to claim 3, characterized in that, The method described above involves stratifying all recognition results based on the ordinate values ​​in the corner coordinates of the recognition results. The recognition results for each layer are sorted according to the x-coordinate values ​​of the corner coordinates, including: Sort all recognition results according to the Y direction to obtain set B; Traverse the set B; Layered sorting steps: Store data items in a pre-established set C, determine whether the absolute value of the coordinate difference between the current data item and the next data item in the Y direction exceeds the layering threshold set in the Y direction. The layering threshold is set according to the pixel interval in the Y direction of each layer of data. If it does not exceed the threshold, continue to repeat this step to add data items to the set C. If the hierarchical threshold is exceeded, the set C is sorted in the X direction, and all sorted data items in the set C are stored in a pre-established set D, and then the set C is cleared. Continue traversing set B, repeating the hierarchical sorting steps, until all data items in set B have been traversed.

5. The panel detection and recognition method according to claim 3, characterized in that, Before fusing the recognition results of the text information with the recognition results of the status information, the method further includes: When the recognition result of the text information or the recognition result of the status information does not match the actual situation, the recognition result of the text information and the recognition result of the status information are processed linearly.

6. The panel detection and recognition method according to claim 3, characterized in that, After fusing the recognition results of the text information and the recognition results of the status information, the method further includes: Based on the recognition result generated after fusion, it is determined whether dynamic information exists; if so, dynamic information detection and recognition are performed. The process of dynamic information detection and recognition includes: The dynamic information is removed from the recognition results by using keywords and then added to the cache list; The specific business module to which the dynamic information belongs is determined by the anchor points of the static information retained in the recognition results.

7. A panel detection and recognition method according to any one of claims 1-6, characterized in that, The process of processing configuration items based on the fused one-dimensional array and business logic definitions, mapping the fused one-dimensional array to the object list containing the processing configuration items, and clustering the recognition results based on the processing configuration items according to the page logic includes: The processing configuration items are parsed into a list of objects, including static information configuration and module-based processing configuration. All the recognition results are mapped to the object list containing the processing configuration items, except for static information which does not need to be mapped. Clustering is performed according to the modular processing configuration.

8. A panel detection and recognition device, characterized in that, include: The first identification unit is used to acquire an image of the device panel and identify the display screen and indicator lights of the device panel from the image of the device panel; The second recognition unit is used to recognize the text information displayed on the display screen using a pre-trained text detection and recognition model, and to recognize the status information displayed on the display screen using image processing technology; wherein, the recognition results of the text information and the status information both include the corresponding information obtained and its relative position information relative to the display screen; The conversion and fusion unit is used to use a hierarchical sorting algorithm to convert the recognition results of the text information and the recognition results of the state information into one-dimensional arrays respectively and fuse them into a complete one-dimensional array. And a parsing unit, configured to process configuration items according to the fused one-dimensional array and business logic definition, map the fused one-dimensional array to the object list where the processing configuration items are located, and perform clustering processing on the recognition results based on the processing configuration items and the page logic situation.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the panel detection and recognition method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the panel detection and recognition method according to any one of claims 1-7.

Images