Panel detection and identification method, apparatus, device, and computer-readable storage medium

Abstract

The application discloses a panel detection and recognition method and device, equipment and a computer readable storage medium, relates to the technical field of panel detection, optical character recognition and the like, and combines panel detection and optical character recognition technology to realize analysis and extraction of panel expression content information and service logic information, and utilizes a hierarchical sorting algorithm to realize structured processing of recognition result data, simultaneously introduces processing configuration item definition and analysis, facilitates display, analysis and data analysis of recognized content, improves the recognition effect, especially can achieve the expected recognition effect for recognizing a page with complex service logic, and makes the panel detection application more simple and convenient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of panel detection, optical character recognition, and the like, and in particular to a panel detection and recognition method, device, equipment, and computer readable storage medium. BACKGROUND

[0002] Panel detection and optical character recognition (OCR) are two important directions in the field of computer vision, each with unique application scenarios and technical implementations, while also being interconnected and complementary in some aspects. Panel detection aims to identify and locate specific panel regions, such as instrument panels, electronic screens, billboards, etc., from images. Its application fields include intelligent monitoring systems, traffic sign recognition in autonomous driving, screen content analysis of smart home devices, etc. The core goal of OCR is to extract readable text information from images. With the development of technology, OCR has been widely applied in document scanning, license plate recognition, bill processing, intelligent typesetting, etc. Its applications are not limited to static images, but also extend to text recognition in video streams.

[0003] However, there is no reliable method for processing OCR recognition results in the field of panel detection, especially for some complex device panels. SUMMARY

[0004] In view of the limitations of existing panel detection technology, the present application proposes a panel detection and recognition method, device, equipment, and computer readable storage medium, which combines panel detection and optical character recognition technology to achieve analysis and extraction of panel expression content information and business logic information, improving the recognition effect.

[0005] The present application is implemented through the following technical solutions:

[0006] A panel detection and recognition method, comprising:

[0007] Obtaining a device panel image and identifying the display screen and indicator light of the device panel from the device panel image;

[0008] Using a pre-trained text detection and recognition model to identify the text information displayed in the display screen, and using image processing technology to identify the state information displayed in the display screen; wherein the recognition results of the text information and the state information each include the corresponding information identified and its relative position information relative to the display screen;

[0009] Using a hierarchical sorting algorithm to convert the recognition results of the text information and the recognition results of the state information into one-dimensional arrays respectively and fuse them to form a complete one-dimensional array;

[0010] The one-dimensional array formed by fusion and the business logic definition processing configuration item are mapped to an object list where the processing configuration item is located, and the recognition result is clustered according to the page logic condition based on the processing configuration item.

[0011] In some embodiments, the pre-trained text detection and recognition model is used to recognize the text information displayed on the display screen, and the state information displayed on the display screen is recognized through image processing technology, including:

[0012] Image preprocessing is performed on the recognized display screen image to generate input vector data;

[0013] The input vector data is input into a text detection model for text detection to obtain a detection result; the detection result is a text detection box;

[0014] The display screen image is cropped according to the detection result to obtain an image list containing only text;

[0015] The image list is preprocessed to generate a vector data list;

[0016] The vector data list is input into a text recognition model for inference, and finally a text recognition result list is obtained; the result list and the detection result together constitute the recognition result of the text information;

[0017] The recognized display screen image is converted to an HSV color space, and then color information representing the state of the display screen is extracted through an OpenCV color extraction algorithm to obtain a recognition result of the state information, including the state information and its relative position information on the display screen.

[0018] In some embodiments, the recognition result of the text information and the recognition result of the state information are converted into one-dimensional arrays respectively by using a hierarchical sorting algorithm, including:

[0019] The corner coordinates of the recognition result are used to represent the recognition result;

[0020] All recognition results are layered according to the vertical coordinate values in the corner coordinates of the recognition result; and the recognition results in each layer are sorted according to the horizontal coordinate values in the corner coordinates of the recognition result;

[0021] The sorted recognition results are concatenated into an ordered one-dimensional array.

[0022] In some embodiments, the all recognition results are layered according to the vertical coordinate values in the corner coordinates of the recognition result; and the recognition results in each layer are sorted according to the horizontal coordinate values in the corner coordinates of the recognition result, including:

[0023] sorting all the recognition results according to the Y direction to obtain a set B;

[0024] traversing the set B;

[0025] a hierarchical sorting step: storing data items in a pre-established set C, judging whether the absolute value of the coordinate difference in the Y direction between the current data item and the next data item exceeds a hierarchical threshold value set in the Y direction, the hierarchical threshold value being set according to the pixel interval in the Y direction of each layer of data, if not, continuing to add data items to the set C in this step;

[0026] if the hierarchical threshold value is exceeded, performing sorting in the X direction on the set C, and storing all the sorted data items in the set C into a pre-established set D, and then emptying the set C;

[0027] continuing to traverse the set B, and repeating the hierarchical sorting step until the traversal of all data items in the set B is completed.

[0028] In some embodiments, before the recognition result of the text information and the recognition result of the state information are fused, the following steps are further included:

[0029] when the recognition result of the text information or the recognition result of the state information does not conform to the actual situation, performing linear processing on the recognition result of the text information and the recognition result of the state information.

[0030] In some embodiments, after the recognition result of the text information and the recognition result of the state information are fused, the following steps are further included:

[0031] judging whether there is dynamic information according to the generated recognition result after fusion, if yes, performing dynamic information detection and recognition; the process of the dynamic information detection and recognition includes:

[0032] eliminating the dynamic information from the recognition result by a keyword and adding it to a cache list;

[0033] judging the specific business module to which the dynamic information belongs through the anchor point of the static information retained in the recognition result.

[0034] In some embodiments, the one-dimensional array formed by fusion and the business logic definition processing configuration item are processed, and the one-dimensional array formed by fusion is mapped to an object list in which the processing configuration item is located, and the recognition result is clustered based on the processing configuration item according to the page logic, including:

[0035] the processing configuration item is parsed into an object list, and the processing configuration item includes static information configuration and sub-module processing configuration;

[0036] mapping all the recognition results to an object list where the processing configuration item is located, wherein static information does not need to be mapped;

[0037] performing clustering processing according to the sub-module processing configuration.

[0038] In a second aspect, the present application provides a panel detection and recognition device, comprising:

[0039] a first recognition unit configured to acquire a device panel image and recognize a display screen and an indicator light of the device panel from the device panel image;

[0040] a second recognition unit configured to recognize text information displayed on the display screen by using a pre-trained text detection and recognition model and recognize state information displayed on the display screen by using an image processing technology; wherein the recognition results of the text information and the state information both include the corresponding information recognized and the relative position information of the corresponding information relative to the display screen;

[0041] a conversion and fusion unit configured to convert the recognition results of the text information and the recognition results of the state information into one-dimensional arrays respectively by using a hierarchical sorting algorithm and fuse the one-dimensional arrays to form a complete one-dimensional array;

[0042] a parsing unit configured to define a processing configuration item according to the one-dimensional array fused and map the one-dimensional array fused to an object list where the processing configuration item is located, and perform clustering processing on the recognition results according to a page logic condition based on the processing configuration item.

[0043] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the embodiments of the panel detection and recognition method described above when executing the computer program.

[0044] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any of the embodiments of the panel detection and recognition method described above.

[0045] The present application provides a panel detection and recognition method, which combines panel detection and optical character recognition technology to realize analysis and extraction of panel expression content information and business logic information, and uses a hierarchical sorting algorithm to realize structured processing of recognition result data, and introduces processing configuration item definition and parsing to facilitate display, analysis and data analysis of recognized content, improve recognition effect, especially for recognizing pages with complex business logic, which can also achieve the expected recognition effect, and make panel detection application more simple and convenient.

[0046] Accordingly, the panel detection and recognition device, computer equipment, and computer-readable storage medium proposed in this application also possess the same technical effects as described above. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of the embodiments of this application and form part of this application, do not constitute a limitation on the embodiments of this application. In the drawings:

[0048] Figure 1 This is a flowchart of the identification method proposed in the embodiments of this application;

[0049] Figure 2 This is a schematic diagram of the analog device panel according to an embodiment of this application;

[0050] Figure 3 for Figure 2 The diagram shows the indicator light recognition results of the simulated device panel.

[0051] Figure 4 for Figure 2 The diagram shows the display screen recognition results of the simulated device panel.

[0052] Figure 5 For the purposes of this application's embodiments Figure 4 The recognition results shown on the display screen;

[0053] Figure 6 The hierarchical sorting algorithm used in this application embodiment is for... Figure 5 The recognition results and processing results are shown below;

[0054] Figure 7 In this embodiment of the application, the corner coordinates of data items on the same layer are not on the same layer;

[0055] Figure 8 for Figure 5 The recognition results shown are the final recognition results after hierarchical sorting and fusion;

[0056] Figure 9 This is a schematic diagram for detecting dynamic information;

[0057] Figure 10 This refers to the definition and parsing results of the processing configuration items in this application embodiment;

[0058] Figure 11 This is a flowchart illustrating the OCR recognition process of an embodiment of this application.

[0059] Figure 12 This is a block diagram illustrating the principle of the identification device proposed in the embodiments of this application;

[0060] Figure 13 This is a schematic diagram of the identification system architecture proposed in an embodiment of this application;

[0061] Figure 14 An electronic device according to an embodiment of the present application;

[0062] Figure 15 A computer readable storage medium according to an embodiment of the present application;

[0063] Reference signs and corresponding names of parts:

[0064] 200 - recognition device, 201 - first recognition unit, 202 - second recognition unit, 203 - conversion fusion unit, 204 - analysis unit, 300 - recognition system, 301 - input device, 302 - output device, 303 - processor A, 304 - memory A, 400 - electronic device, 410 - memory B, 420 - processor B, 411 - computer program A, 500 - computer readable storage medium, 511 - computer program B. DETAILED DESCRIPTION

[0065] Hereinafter, the term "include" or "may include" used in various embodiments of the present application indicates the existence of the invented function, operation, or element, and does not limit one or more functions, operations, or elements from being added. Also, as used in various embodiments of the present application, the terms "include", "have", and their conjugates merely indicate the presence of the mentioned features, numbers, steps, operations, elements, components, or combinations thereof, and should not be construed as excluding the possibility of the addition of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0066] In various embodiments of the present application, the expression "or" or "at least one of A or / and B" includes any combination of the listed terms or all the terms. For example, the expression "A or B" or "at least one of A or / and B" can include A, can include B, or can include both A and B.

[0067] The expressions (such as "first", "second", etc.) used in various embodiments of the present application can modify various constituent elements in various embodiments, but can not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the described elements. The above expressions are used only for the purpose of distinguishing one element from another element. For example, the first user device and the second user device indicate different user devices, although both are user devices. For example, a first element can be referred to as a second element, and likewise, a second element can be referred to as a first element, without departing from the scope of various embodiments of the present application.

[0068] It should be noted that if a description connects one component element to another component element, the first component element can be directly connected to the second component element, and a third component element can be "connected" between the first component element and the second component element. Conversely, when one component element is "directly connected" to another component element, it is understood that there is no third component element between the first component element and the second component element.

[0069] The terms used in the various embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the various embodiments of the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having a meaning that is the same as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized or overly formal meaning unless clearly defined in the various embodiments of the present application.

[0070] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description of the present application is made below in combination with embodiments and drawings, the illustrative embodiments of the present application and their descriptions are only used to explain the present application and do not limit the present application.

[0071] The embodiment of the present application proposes a panel detection and recognition method, which combines optical character recognition and panel detection technology to realize general device panel detection. The method can be applied to all panel detection application scenarios, can improve the recognition accuracy, and can also recognize pages with complex business logic, achieving the expected recognition effect.

[0072] As shown in Figure 1 The panel detection and recognition method proposed by the embodiment of the present application includes the following steps:

[0073] Step 1, obtaining a device panel image and identifying related main elements of the device panel from the device panel image; wherein the identified related main elements include but are not limited to display screen, indicator light and the like;

[0074] Step 2, identifying the text information displayed in the display screen by using a pre-trained text detection and recognition model, and identifying the state information displayed in the display screen by using image processing technology;

[0075] Step 3, converting the text information recognition result and the state information recognition result into one-dimensional arrays respectively by using a hierarchical sorting algorithm and fusing to form a complete one-dimensional array;

[0076] Step 4, processing the configuration item according to the fused one-dimensional array and the business logic definition, mapping the fused one-dimensional array to an object list where the processing configuration item is located, and performing clustering processing on the recognition result according to the page logic case based on the processing configuration item.

[0077] Further, the step 1 of the embodiment of the present application is implemented as follows:

[0078] Device panel image acquisition, the embodiment of the present application uses a camera to realize real-time acquisition of the device panel image, and the following requirements are required for the acquisition device environment: (1) camera configuration, first, a high-resolution industrial camera is required, on the one hand, the camera needs to work continuously to ensure the practicability and reliability of the camera, on the other hand, the text detection needs a clear picture, especially the clear text information needs to be captured to improve the accuracy of OCR recognition, and the color of the captured picture is clear to facilitate the recognition of the signal light; second, the camera needs to support common parameter configuration such as camera exposure, brightness, contrast, focal length, etc., to improve the quality of the camera captured picture through parameter configuration; (2) image acquisition configuration, image acquisition can use video stream, shooting and other modes, which are selected according to specific business needs, video stream is used for high real-time requirement, shooting is used for low real-time requirement; in addition, CPU is used for low real-time requirement, and GPU is introduced for high real-time requirement for OCR model inference; (3) light effect, when the image acquisition frequency is low, dynamic parameter configuration is used, that is, two different exposure parameters (low exposure parameter and high exposure parameter) are used in different time periods to realize the shooting and acquisition of the device panel; when the image acquisition frequency is high, a set of low exposure static parameters (low exposure parameter) can be used to realize the shooting and acquisition of the device panel, in the case of low exposure, the indicator light and display screen of the device panel can be clearly captured, the disadvantage is that only the display screen and the device panel indicator light can be identified, and other information of the device panel cannot be extracted, which is selected according to the actual application scene; in addition, the shooting light needs to be as uniform as possible to avoid reflection and shadow; (4) installation condition, keep fixed state, require the camera and the device to be relatively fixed, including the distance between the camera and the device, the rotation angle of the camera, etc.; the working environment temperature, humidity and other working environment of the camera and the device match, which does not affect the normal maintenance and use of the device.

[0079] Identifying the device indicator light from the acquired device panel image, wherein the device indicator light refers to the signal light on the device panel that can be used to judge some state, including the on-off state, connection state, running state, self-checking state and other all extractable state information. The device indicator light can be distinguished by different colors and brightness, and the relative position of the device indicator light also needs to be identified to distinguish the indicator light representing different meanings. Figure 2 The indicator light of the simulated device panel is shown as Figure 3The identification process of the device indicator light is as follows:

[0080] Image conversion: Convert the original RGB image to HSV space, which is the core of converting the superposition table of red, green, and blue channels in the RGB image to hue (H), saturation (S), and brightness (V) three-dimensional representation in HSV space. First, the values of the RGB three channels need to be standardized to the range of [0, 1], that is, the color channel value is divided by 255. The conversion formula is as follows

[0081]

[0082] where represent the red, green, and blue coordinates, respectively, and their values are real numbers between 0 and 1; , indicating the maximum value; , indicating the minimum value; represent the values of hue (H), saturation (S), and brightness (V), respectively.

[0083] Color threshold screening: According to the color of the target indicator light, determine its threshold range in HSV space, for example, the HSV range of a red indicator light may be [340°, 0.6, 0.6] to [20°, 1, 1], that is, the H value is between 340°~20°, the S value is between 0.6~1, and the V value is between 0.6~1. Through color threshold screening, the relevant color area of the panel is screened out and binarization processing is performed to generate a binary mask image.

[0084] Morphological processing: Perform erosion and dilation operations on the binary mask image to remove noise and small area interference regions and find the connected region contour.

[0085] Indicator light identification: According to the area, circularity, and other characteristics of the connected region contour, screen out the region that meets the indicator light form, locate the screened region, and perform state judgment of the indicator light.

[0086] At the same time, the device display screen is identified from the obtained device panel image. The identification of the device display screen is to provide the minimum image recognition range for the display screen content recognition OCR. In general, directly using the display screen for recognition can improve the accuracy of recognition and improve the video frame processing speed. The device display screen recognition is only performed at the beginning of the video processing process. After the display screen recognition obtains the relative position information of the display screen relative to the device panel, the relative position information is put into the cache, and the display screen position is directly obtained from the cache during processing. The processing process after the original device panel is cropped. Figure 2 The display screen after the simulation device panel is cropped is shown in Figure 4 The identification process of the device display screen is as follows:

[0087] Device panel image gray processing: a contour tracking algorithm based on topology is adopted, and the device panel image is first processed into a gray image;

[0088] Gaussian blur processing: the obtained gray image is subjected to Gaussian blur processing;

[0089] Edge detection: the image subjected to Gaussian blur processing is subjected to edge detection, and the Canny edge detection method can be adopted;

[0090] Closed edge: the edge detection result is subjected to edge closing;

[0091] Contour extraction: all contours are found from the closed edge;

[0092] The contours meeting the conditions are screened out, and the device display screen is obtained by cropping the device panel image.

[0093] In the embodiment of the application, all contour information of the device panel image is obtained through the contour extraction process, the contours meeting the device display screen are screened out through the contour size, and finally the device display screen is obtained by cropping the device panel image through the contour line.

[0094] Further, in step 2, the content displayed in the device display screen mainly includes text information, state information and dynamic information.

[0095] Most of the information in the device display screen is text information, and the state of the text information displayed in the display screen has three types, namely static text, variable text and dynamic text. The static text is the interface display title, table header and other information, which will not change in the device display screen, that is, it is fixed. The variable text is the text information that will be displayed in the device display screen, such as Figure 4 As shown in the device display screen of the D module, the working state can be a fault or normal, but will not be displayed. The dynamic text is text information that can be displayed or not displayed in the device display screen. The static text can not be processed in the recognition, and a data dictionary can be directly established. The variable text and the dynamic text must be recognized by OCR, and the recognition process needs to be distinguished. The OCR recognition mainly uses a pre-set text detection and recognition model to perform model reasoning on the device display screen image, and the OCR recognition process is as shown in Figure 11The text detection recognition model is inferred by using a PaddleInference library. First, the text detection model and the text recognition model are loaded by using the PaddleInference library, and then the input device display screen image is subjected to image preprocessing operations such as image size adjustment, normalization, channel conversion, and the like to obtain vector data input to the text detection model, the vector data is input to the text detection model to obtain a detection result, the device display screen image is cropped to obtain an image list containing only text by using the text detection result, i.e., a text detection box, and the image list is subjected to preprocessing operations such as image size adjustment, normalization, channel conversion, and the like to obtain a vector data list for text recognition, the vector data list is input to the text recognition model for inference, and finally a result list for text recognition is obtained.

[0096] The state information refers to information in the device display screen that can be used to describe some state, in addition to the text information. For example, the brightness of the display screen can be used as a basis for the device power-on and power-off when the device does not have a power switch state indicator light. For example, the state of each module related to the business, different colors represent different business meanings such as Figure 4 The state information detection and recognition uses the HSV color space and is implemented by using an image processing library. First, the elements representing the state information in the display screen are analyzed, and it is determined whether different colors are used to distinguish different states. Then, the color detection is implemented by using the library provided by OpenCV.

[0097] The specific steps are as follows: (1) converting the image to the HSV format, (2) setting the color range, (3) image binarization, (4) contour recognition, and (5) calculating the detection box. The detection box indicates that the relevant pixel positions in the image exist the color to be detected, and the specific meaning of the corresponding business is obtained. The detection box of the detection result is similar to the detection box of the text detection, which indicates the specific pixel position of the detection information in the image.

[0098] The dynamic information represents the pop-up box that appears in the device display screen during the running process, and the prompt summary information, such as device self-checking. The self-checking result is displayed in the display screen, and the pop-up box can disappear after a few seconds. The camera needs to capture this state and perform recognition. The dynamic information detection usually contains text information or color information. If it is text, the OCR is used for detection and recognition, and then the text matching is performed for processing. If it is color, the HSV color space is used for detection. The dynamic information detection is independent of the above detection process.

[0099] Further, in step 3 of the embodiment of the present application, the OCR recognition result is a set of OCR data items (text detection result + confidence + detection box). Regardless of the device display screen itself, the distribution of the OCR recognition result is as follows: Figure 5As shown, it is a two-dimensional planar distribution in the horizontal (X) and vertical (Y) directions. The Y-axis represents the height of the image to be detected (i.e., the image on the device display screen), and the X-axis represents the width of the image to be detected. The figure shows that the OCR recognition results are distributed in a two-dimensional plane on the device display screen. Assuming these recognition results are represented by their corner points, for example... Figure 5 The point P(x, y) in the upper left corner represents these recognition results. Using the positional information of these results in the image, the recognition results can be converted into a one-dimensional array arranged according to a certain pattern. First, the data can be layered in the Y direction, that is, layered according to the y coordinate of point P (specifically, within a certain height range of the image, there are multiple recognition results; after sorting the detection results along the Y axis, traversing the set, the data in different layers will show obvious gradient changes; for example, the y-value change of the detection results in the same layer is generally within 10 pixels, but when traversing to a new element, the y-value change of the pixel exceeds 10 pixels, indicating that the next layer has been reached; that is, data layering is the differentiation of detection results according to the pixel height of the image). Then, the data is sorted in the X direction, that is, the layered data is sorted according to the x coordinate of point P, resulting in an ordered one-dimensional array in the X and Y directions, as shown below. Figure 6 The results shown are sorted by layering in the Y direction and sorting in the X direction and then concatenated into an ordered one-dimensional array.

[0100] Specifically, the algorithm for hierarchically sorting the recognition results is as follows:

[0101] The data items in the OCR recognition result set A are sorted according to the Y direction to obtain set B. This step aims to make the entire detection result ordered by sorting by the y coordinate, so that the set can be traversed sequentially and the data from the same layer can be determined according to the set threshold.

[0102] Traverse set B;

[0103] Hierarchical sorting steps: Store data items in a pre-established set C, and determine whether the absolute value of the difference between the y-coordinate of the current data item and the next data item (for example, calculate the absolute value of the difference using the y-coordinate of the upper left corner of the detection box) exceeds the hierarchical threshold set in the Y direction. This hierarchical threshold is set based on experience. If it does not exceed the threshold, continue to add data items to set C in a loop.

[0104] If the hierarchical threshold is exceeded, sort set C in the X direction, store the sorted data items of set C into a pre-established set D, and then clear set C.

[0105] Continue iterating through set B, repeating the hierarchical sorting steps described above, until all data items in set B have been traversed.

[0106] Further, when layering, the angle point (for example, the corner point) between the data items of each layer dataset is not a fixed value, but fluctuates up and down, so the layering threshold in the layering sorting algorithm of the recognition result dataset must be greater than the maximum amplitude of the fluctuation, so as to distinguish the levels of the data, thereby obtaining a useful one-dimensional array.

[0107] The state information recognition result is similar to the text information recognition result, and includes recognition result information and relative position information relative to the display screen, so the two-dimensional data result is also converted into a one-dimensional array according to the Y-direction layering and X-direction sorting algorithm.

[0108] If the text information recognition result and the state information recognition result are directly fused, there may be problems such as Figure 7 As shown in the figure, the y coordinates of the two points P and Q at the top left corner of the detection box are not in the same layer, but the detection box represented by the two points P and Q should be in the same layer. If the y coordinates are directly used for layering, they may not be in the same layer. In view of this, the present application embodiment introduces P1 and Q1 points (i.e. the coordinates of the lower left corner of the detection box) for linear processing, calculates the middle value of the y coordinates of P and P1 points and Q and Q1 points, so as to divide the detection results represented by P and Q into the same layer. The coordinate conversion formula is as follows:

[0109]

[0110]

[0111] Among them, , P and Q are the converted coordinate points of P and Q respectively, are the y coordinates of P and P1 respectively, are the y coordinates of Q and Q1 respectively, are the x coordinates of P and Q respectively.

[0112] At this time, the coordinates of all the top left corner points of the detection box need to be transformed into the point where the middle line is located, and then layering and sorting can be performed.

[0113] The text information recognition result and the state information recognition result converted into a one-dimensional array are fused to form a complete one-dimensional array, and the meaning of each index in the array is known, Figure 8 The figure shows the result of the recognition result after layering and sorting and fusion.

[0114] If the device display screen includes dynamic text or involves dynamic information, it can be judged by the OCR recognition result, so as to Figure 8 Taking the recognition result shown in the figure as an example, the OCR detection result of the static and dynamic text is 20 items. If dynamic text appears or the number of pop-up box detection is not equal to 20 items, the dynamic information is processed separately.

[0115] Dynamic information detection needs to be analyzed in combination with the specific business logic of the display screen. For dynamically displayed information (i.e., text or indicator lights that may or may not be displayed in a fixed position in the display screen), the text needs to be first removed from the recognition result through a keyword, to ensure that the order logic of the static display information detected through OCR is correct, and then the anchor point of the static information is used to determine the specific business module to which the dynamic information belongs. For dynamically displayed indicator lights, the anchor point of the static information is directly used for judgment, to Figure 4 For example, the device display screen shown in Figure 9 In the detection schematic diagram of the dynamic text shown in FIG. 8, the A module includes the dynamically displayed text "Connection Status: TRUE". It is assumed that the device is not inserted into the network, and the text will not be displayed. After the network cable is inserted, the connection success or failure is displayed according to the connection status. The dynamic display information is added to the OCR recognition result, and all the previous indexes will change. To ensure the hierarchical ordering relationship of the previous static display information, the dynamic information needs to be first removed, for example, through the "connection status" keyword, the dynamic information item in the OCR recognition result list is removed and added to the cache list, and then the display of the dynamic information in the cache list is processed. Since the A module, the B module, and the C module have the same structure, to accurately determine whether the "connection status: TRUE" belongs to the A module, the B module, or the C module, the anchor point of the static information needs to be determined to determine which module the dynamic information belongs to. For the current panel, the point at the top left corner of the text detection box of the A module can be used as the anchor point, and a given pixel range is used to determine whether the "connection status: TRUE" is within the pixel range. If it is, it indicates that the dynamic display information belongs to the "A module".

[0116] Further, through the above steps, the panel content detection and recognition can be realized, and the structured business data can be organized and generated, but the post-processing method is not very friendly, and therefore, the index needs to be created one by one to assign values, and therefore, the step 4 of the embodiment of the present application introduces a configuration item definition and parsing processing. Based on the configuration item, the recognition result can be clustered according to the page business logic, the flexibility of processing is increased, and the processing flow is simplified. The specific process is as follows:

[0117] Processing configuration item definition: as shown in Figure 10 The processing configuration item combines the recognition result and the business logic to perform configuration including static information processing configuration, module processing configuration, and the like. The processing configuration item defines the processing logic of the program.

[0118] Configuration item analysis process: after obtaining the processing configuration item, it is parsed into an object list, first all the recognition results are mapped to the object list where the configuration item is located, and then the configuration is processed according to the sub-module, that is, the A, B, C, D and E business modules are classified and processed through the digital identifier. At this time, the data has a preliminary logic and can be used for display and data analysis.

[0119] The data can be further filled into specific objects, and Figure 4 A, B and C are classified as a class, and D and E are classified as a class in the device display screen shown in the figure. Two for loops can be used to complete object assignment, avoiding individual index assignment. This processing method can be adapted to different data processing scenarios.

[0120] Further, in the embodiment of the application, the PaddleOCR deep learning model is selected as the basic model framework for panel detection. However, the model is a general model suitable for all optical character recognition scenarios, but the recognition effect may be poor for specific scenarios. For example, in the application of liquid crystal display screen recognition, the text detection and recognition effect of the device display screen using the PaddleOCR deep learning model cannot achieve the expected effect. In view of this, the PaddleOCR deep learning model is strengthened in the embodiment of the application as the final text detection and recognition model. The specific training process is as follows:

[0121] Data set construction: capture the device panel images that need to be recognized by the camera, and use different light, different distance, different display content and other strategies to obtain a number of device real images (for example, about 50). The number of image samples for each strategy is evenly distributed. Then the data samples are enhanced, different color image samples are augmented by contrast, saturation, color temperature, and different clarity image samples are augmented by adding image noise, to finally generate more device panel images (for example, about 100).

[0122] Use a semi-automatic labeling tool to label the data, divide the labeled data set into a text detection data set and a text recognition data set, and divide each data set into a training set, a test set and a validation set. When using the tool to divide the data set, the proportions of the training set, the test set and the validation set are set to 6:2:2. The above content completes the data set production.

[0123] Selection of training model: the pre-training weight suitable for the server is selected as the pre-training weight of the text detection model to improve the accuracy of text detection, and the lightweight model weight is selected as the pre-training weight of the text recognition model.

[0124] Configuration and training: configure the dataset, configure the number of training times, configure whether to enable GPU according to the hardware environment, train the model, adjust the parameters if the expected result cannot be achieved, adjust the dataset and repeat the training until the expected result is achieved.

[0125] Based on the same technical concept, the embodiment of the present application also provides a panel detection and recognition device, as shown in Figure 12 The recognition device 200 includes:

[0126] The first recognition unit 201 is configured to acquire a device panel image and recognize relevant main elements of the device panel from the device panel image. The specific implementation manner is as described in step 1 above, and will not be described here again.

[0127] The second recognition unit 202 is configured to recognize text information displayed on the display screen by using a pre-trained text detection and recognition model, and recognize state information displayed on the display screen by using an image processing technology. The specific implementation manner is as described in step 2 above, and will not be described here again.

[0128] The conversion and fusion unit 203 is configured to convert the text information recognition result and the state information recognition result into one-dimensional arrays respectively by using a hierarchical sorting algorithm, and fuse the one-dimensional arrays to form a complete one-dimensional array. The specific implementation manner is as described in step 3 above, and will not be described here again.

[0129] In addition, the analysis unit 204 is configured to process a configuration item according to a business logic definition, map the fused one-dimensional array to an object list in which the configuration item is located, and perform clustering processing on the recognition result based on the configuration item according to a page logic condition. The specific implementation manner is as described in step 4 above, and will not be described here again.

[0130] Further, the recognition device 200 of the embodiment of the present application further includes:

[0131] The model training unit is configured to use a PaddleOCR deep learning model as a basic model framework for panel detection, and train to obtain a final text recognition and detection model.

[0132] Based on the same technical concept, the embodiment of the present application also provides a panel detection and recognition system, as shown in Figure 13 The recognition system 300 provided by the embodiment of the present application includes:

[0133] The input device 301, the output device 302, the processor A 303 and the memory A 304; wherein the number of the processor A 303 and the memory A 304 can be one or more, Figure 13The following description uses a processor A303 and a memory A304 as an example. The input device 301, output device 302, processor A303, and memory A304 can be connected via a bus or other means. Figure 13 Taking the example of a connection between China and Israel via a bus.

[0134] Specifically, by calling the operation instructions stored in memory A304, processor A303 executes the following steps:

[0135] Acquire an image of the device panel and identify the main elements of the device panel from the image; wherein the main elements identified include, but are not limited to, the display screen, indicator lights, etc.

[0136] The text information displayed on the screen is identified using a pre-trained text detection and recognition model, and the status information displayed on the screen is identified using image processing technology.

[0137] A hierarchical sorting algorithm is used to convert the text information recognition results and the state information recognition results into one-dimensional arrays respectively and then merge them to form a complete one-dimensional array.

[0138] The fused one-dimensional array and business logic definition are used to process configuration items, and the fused one-dimensional array is mapped to the object list where the processing configuration items are located. Based on the processing configuration items, the recognition results are clustered according to the page logic.

[0139] Optionally, by calling the operation instructions stored in memory A304, processor A303 is also used to execute any of the embodiments in the corresponding examples of the above identification method.

[0140] Based on the same technical concept described above, this application also proposes an electronic device, such as... Figure 14 As shown, the electronic device 400 includes: a memory B410, a processor B420, and a computer program A411 stored in the memory B410 and executable on the processor B420. When the processor B420 executes the computer program A411, it performs the following steps:

[0141] Acquire an image of the device panel and identify the main elements of the device panel from the image; wherein the main elements identified include, but are not limited to, the display screen, indicator lights, etc.

[0142] The text information displayed on the screen is identified using a pre-trained text detection and recognition model, and the status information displayed on the screen is identified using image processing technology.

[0143] A hierarchical sorting algorithm is used to convert the text information recognition results and the state information recognition results into one-dimensional arrays respectively and then merge them to form a complete one-dimensional array.

[0144] According to the one-dimensional array formed by fusion and the processing configuration item defined by business logic, the one-dimensional array formed by fusion is mapped to an object list where the processing configuration item is located, and the recognition result is clustered according to the page logic condition based on the processing configuration item.

[0145] Optionally, when the processor B 420 executes the computer program A 411, any implementation in the corresponding embodiments of the above-mentioned recognition method can be realized.

[0146] It should be noted that the electronic device proposed in the embodiments of the present application is a device used to implement the above-mentioned recognition method, and therefore based on the above-mentioned recognition method proposed in the embodiments of the present application, those skilled in the art can understand the specific implementation of the electronic device of the embodiments of the present application and its various forms of changes, so here the specific implementation of the electronic device to implement the above-mentioned recognition method is not introduced in detail, as long as the electronic device used to implement the above-mentioned recognition method is implemented by those skilled in the art, it belongs to the scope of protection of the present application.

[0147] Based on the same technical concept, the embodiments of the present application also propose a computer readable storage medium, as shown in the figure, the computer readable storage medium 500 stores a computer program B 511, and the computer program B 511 is executed by a processor to realize the following steps: Figure 15

[0148] Obtain a device panel image, and identify the relevant main elements of the device panel from the device panel image; wherein the identified relevant main elements include but are not limited to display screen, indicator light and other elements;

[0149] Use a pre-trained text detection and recognition model to identify the text information displayed on the display screen, and use image processing technology to identify the state information displayed on the display screen;

[0150] Use a hierarchical sorting algorithm to convert the text information recognition result and the state information recognition result into one-dimensional arrays respectively and form a complete one-dimensional array by fusion;

[0151] According to the one-dimensional array formed by fusion and the processing configuration item defined by business logic, the one-dimensional array formed by fusion is mapped to an object list where the processing configuration item is located, and the recognition result is clustered according to the page logic condition based on the processing configuration item.

[0152] Optionally, the computer program B 511 can realize any implementation in the corresponding embodiments of the above-mentioned recognition method when executed by the processor.

[0153] ​It should be noted that the description of each of the above-described embodiments has been presented for purposes of illustration and description, and is not intended to be exhaustive or to limit the embodiments to the embodiments described. Numerous additional embodiments will be apparent to those skilled in the art in view of this disclosure.

[0154] Those skilled in the art will appreciate that embodiments of the application can be readily used for providing methods, apparatuses, and computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0155] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function of one or more of the steps in the flowchart illustrations and / or block diagrams.

[0156] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function of one or more of the steps in the flowchart illustrations and / or block diagrams.

[0157] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1 means for performing the function of one or more of the steps in the flowchart illustrations and / or block diagrams.

[0158] The above detailed description of the specific implementation is further detailed for the purpose of the application, technical solutions and beneficial effects, and it should be understood that the above description is only for the specific implementation of the application and is not used to limit the protection scope of the application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A panel detection identification method, characterized in that, The method comprises the following steps: acquiring a device panel image and identifying a display screen and an indicator light of the device panel from the device panel image; identifying text information displayed in the display screen by using a pre-trained text detection and recognition model and identifying state information displayed in the display screen by using an image processing technique; wherein the identification results of the text information and the state information each comprise corresponding information and relative position information of the corresponding information relative to the display screen; converting the identification results of the text information and the identification results of the state information into one-dimensional arrays respectively by using a hierarchical sorting algorithm and fusing the one-dimensional arrays to form a complete one-dimensional array; processing a configuration item according to the fused one-dimensional array and a business logic definition, mapping the fused one-dimensional array to an object list in which the configuration item is located, and performing clustering processing on the identification results based on the configuration item according to a page logic condition; the step of converting the identification results of the text information and the identification results of the state information into one-dimensional arrays respectively by using a hierarchical sorting algorithm comprises the following steps: using corner point coordinates of the identification results to represent the identification results; performing hierarchical processing on all the identification results according to longitudinal coordinate values in the corner point coordinates of the identification results and performing sorting on the identification results of each layer according to transverse coordinate values in the corner point coordinates of the identification results; specifically, sorting all the identification results in the Y direction to obtain a set B; traversing the set B; hierarchical sorting step: storing data items in a pre-established set C; judging whether an absolute value of a coordinate difference in the Y direction between a current data item and a next data item exceeds a hierarchical threshold value in the Y direction, wherein the hierarchical threshold value in the Y direction is set according to a pixel interval in the Y direction of each layer of data; if not, continuing to add data items to the set C in the hierarchical sorting step; if yes, performing sorting in the X direction on the set C, storing all the sorted data items in the set C in a pre-established set D, and then emptying the set C; continuing to traverse the set B and repeating the hierarchical sorting step until the traversal of all the data items in the set B is completed; concatenating the sorted identification results into an ordered one-dimensional array.

2. The method of claim 1, wherein, the step of identifying the text information displayed in the display screen by using a pre-trained text detection and recognition model and identifying the state information displayed in the display screen by using an image processing technique comprises the following steps: performing image preprocessing operations on the identified display screen image to generate input vector data; inputting the input vector data into a text detection model to perform text detection and obtain a detection result; the detection result is a text detection frame; cropping the display screen image according to the detection result to obtain an image list containing only text; performing preprocessing operations on the image list to generate a vector data list; inputting the vector data list into a text recognition model to perform inference, and finally obtaining a result list of text recognition; the result list and the detection result jointly constitute the identification result of the text information; The identified display screen image is converted to an HSV color space, and image processing is performed to obtain the recognition result of the state information, including state information and position information thereof relative to the display screen.

3. The method of claim 1, wherein the step of detecting the panel comprises: Before the recognition result of the text information and the recognition result of the state information are fused, the method further includes: When the recognition result of the text information or the recognition result of the state information does not match the actual situation, linear processing is performed on the recognition result of the text information and the recognition result of the state information.

4. The method of claim 1, wherein, After the recognition result of the text information and the recognition result of the state information are fused, the method further includes: According to the recognition result generated after fusion, it is determined whether dynamic information exists, and if so, dynamic information detection and recognition are performed; the process of the dynamic information detection and recognition includes: The dynamic information is removed from the recognition result through a keyword and is added to a cache list; Anchors of static information retained in the recognition result are used to determine the specific business module to which the dynamic information belongs.

5. The method of claim 1-4, wherein, The one-dimensional array formed by fusion and the processing configuration item defined according to business logic are processed, and the one-dimensional array formed by fusion is mapped to an object list in which the processing configuration item is located, and the recognition result is clustered based on the processing configuration item according to page logic, including: The processing configuration item is parsed into an object list, and the processing configuration item includes static information configuration and submodule processing configuration; The recognition result is all mapped to the object list in which the processing configuration item is located, and static information does not need to be mapped; The clustering processing is performed according to the submodule processing configuration.

6. A panel detection recognition apparatus, characterized by, including: A first recognition unit is configured to acquire a device panel image and recognize a display screen and an indicator light of a device panel from the device panel image; A second recognition unit is configured to recognize text information displayed in the display screen by using a pre-trained text detection and recognition model and recognize state information displayed in the display screen by using an image processing technique; wherein the recognition result of the text information and the recognition result of the state information both include the recognized corresponding information and relative position information thereof relative to the display screen; A conversion and fusion unit is configured to convert the recognition result of the text information and the recognition result of the state information into one-dimensional arrays respectively by using a hierarchical sorting algorithm and fuse the one-dimensional arrays to form a complete one-dimensional array; and an analysis unit is configured to define a processing configuration item according to the one-dimensional array formed by fusion and business logic and map the one-dimensional array formed by fusion to an object list in which the processing configuration item is located, and cluster the recognition result based on the processing configuration item according to page logic; The hierarchical sorting algorithm is used to convert the recognition result of the text information and the recognition result of the state information into one-dimensional arrays, including: The corner point coordinates of the recognition result are used to represent the recognition result; According to the longitudinal coordinate value in the corner point coordinate of the identification result, all the identification results are layered, and according to the horizontal coordinate value in the corner point coordinate of the identification result, the identification results in each layer are sorted, specifically: all the identification results are sorted according to the Y direction to obtain a set B; the set B is traversed; the layering and sorting step: a data item is stored in a pre-established set C, it is judged whether the absolute value of the coordinate difference in the Y direction between the current data item and the next data item exceeds the layering threshold value set in the Y direction, the layering threshold value is set according to the pixel interval of the Y direction of each layer of data, if not, the step is continued to add the data item to the set C; if the layering threshold value is exceeded, the set C is sorted in the X direction, and the sorted data items in the set C are all stored in a pre-established set D, and then the set C is emptied; the set B is continuously traversed, and the layering and sorting step is repeated until the traversal of all the data items in the set B is completed; The sorted identification results are concatenated into an ordered one-dimensional array. 7.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to realize the panel detection and identification method in any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the panel detection and identification method in any one of claims 1-5.

Images