Interface judgment method and system based on image text recognition after SCPI issuing

Abstract

The application discloses a method and system for interface judgment after SCPI issuing based on image text recognition. The method comprises: obtaining test task data and issuing target SCPI command to the device under test; continuously obtaining interface images after the command issuing, judging whether the interface is stable based on the pixel difference of adjacent images; obtaining target interface image after the interface is stable, determining semantic anchor points through text recognition, and determining target recognition area by combining normalized offset and current resolution; extracting features such as text, color, structural similarity and local difference in the target area; generating interface judgment result according to interface features, expected interface result and judgment rules, and supplementing judgment in a rule way when the first judgment result is not reliable, and finally outputting test conclusion and report. The application can improve the accuracy of interface display state judgment in SCPI automatic test.

Description

Technical Field

[0001] This application relates, and particularly to, a method and system for determining the interface after SCPI is issued based on image text recognition. Background Technology

[0002] SCPI, as a commonly used standard command set for electronic measuring instruments, is widely used in the remote control and automated testing of devices such as oscilloscopes, signal generators, and multimeters. Current SCPI testing typically employs a command issuance and result query approach, determining command execution success by reading parameters or status information returned by the instrument. However, in practical applications, even if the command returns normally, issues may arise such as outdated interface updates, incorrect parameter display, abnormal mode identification, or missing alarm prompts. Furthermore, some functional states can only be visually displayed through the instrument's UI and cannot be directly obtained through SCPI command queries. Existing technologies rely heavily on manual observation for interface status verification, resulting in low automation, limited judgment dimensions, low efficiency, and poor traceability. Therefore, a technical solution is needed that can automatically determine the instrument interface status after SCPI command issuance.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] To address the problems in existing technologies, such as the difficulty in confirming whether the interface of the device under test (DUT) has been truly updated by relying solely on the return status of SCPI commands, the low testing efficiency of fixed waiting methods, the inability of fixed-coordinate screenshots to adapt to changes in resolution or interface layout, and the difficulty in automatically judging abnormalities in interface text, color, control status, or waveform display, this application provides a method and system for interface judgment after SCPI issuance based on image text recognition. This method does not judge the test result solely based on the command status returned by the DUT after the target SCPI command is issued, but further acquires the interface image data of the DUT and automatically judges whether the interface of the DUT meets expectations through interface stability judgment, semantic anchor point positioning, target area image extraction, interface feature recognition, and a hierarchical judgment mechanism.

[0005] Firstly, this application provides a method for interface judgment after SCPI issuance based on image text recognition, the method comprising: Acquire test task data, which includes the target SCPI command, expected interface results, and interface judgment rules; and send the target SCPI command to the device under test. After the target SCPI command is sent, continuous interface image data of the device under test is acquired at preset time intervals, and pixel differences are calculated for interface image data of two adjacent frames to obtain interface change ratio data. The interface change ratio data is compared with a preset stability threshold. When the interface change ratio data is less than the preset stability threshold, the interface of the device under test is determined to be in a stable state, and the target interface image data is acquired. When the interface of the device under test is not determined to be in a stable state within a preset timeout period, interface timeout abnormal data is generated. The target interface image data is subjected to text recognition to obtain interface text data and text position data corresponding to the interface text data. Candidate anchor text is determined from the interface text data according to preset semantic anchor data, and spatial rationality is verified according to the text position data of the candidate anchor text to obtain target semantic anchor data. Based on the target semantic anchor point data, the preset normalized offset data and the current interface resolution data, the target recognition region coordinate data is calculated, and the target region image data is extracted from the target interface image data based on the target recognition region coordinate data. Feature extraction is performed on the target region image data to obtain interface feature data, which includes at least one of text feature data, color feature data, structural similarity feature data, and local difference feature data. Based on the interface feature data, the expected interface result, and the interface judgment rule, a first interface judgment result is generated using a first judgment method; when the first interface judgment result does not meet the preset confidence condition, a second judgment method is used to compare the interface feature data with the expected interface result to generate a second interface judgment result. Based on the judgment result of the first interface or the judgment result of the second interface, test conclusion data is generated, and test report data including the test conclusion data, the target interface image data, the interface feature data, and log data is output.

[0006] This method, after the target SCPI command is issued, determines the stable state of the device under test's interface through continuous interface image data. Once the interface stabilizes, semantic anchor points are determined based on text recognition results. Then, normalized offset data and current interface resolution data are combined to determine the target recognition region. Subsequently, features such as text, color, structural similarity, and local differences are extracted from the target recognition region. Test conclusions are generated based on the extracted interface feature data, expected interface results, and interface judgment rules. Therefore, this method can perform image-level verification of the actual interface display state of the device under test beyond the SCPI command return status, reducing misjudgments caused by incomplete interface refresh, changes in interface element positions, or failure of a single recognition method, thus improving the accuracy of interface judgment in automated SCPI testing.

[0007] In some embodiments, calculating the pixel difference between two adjacent frames of interface image data to obtain interface change ratio data includes: Get the current frame's interface image data and the previous frame's interface image data; The pixels at corresponding positions in the current frame interface image data and the previous frame interface image data are compared. When the color channel difference of a pixel at a corresponding position meets a preset difference condition, the pixel is identified as a changed pixel. The interface change ratio data is obtained by comparing the number of changed pixels with the total number of pixels in the interface image data.

[0008] In this way, the interface refresh process can be transformed into a change ratio data, allowing the processor to determine the waiting time based on the actual interface changes. Compared with a fixed delay method, this helps to reduce misjudgments caused by the interface not being fully refreshed and reduces unnecessary waiting time.

[0009] In some embodiments, the step of performing spatial rationality verification based on the text position data of the candidate anchor text to obtain target semantic anchor data includes: Obtain the current position data and historical anchor point position data of the candidate anchor point text; Calculate displacement data based on the current location data and the historical anchor point location data; When the displacement data meets the preset position constraint conditions, the corresponding candidate anchor point text is determined as the valid anchor point text; When multiple valid anchor texts exist, the target semantic anchor data is determined from the multiple valid anchor texts based on the text recognition confidence score.

[0010] In this way, when there are multiple identical or similar characters in the interface, the probability of identifying erroneous text as anchor points can be reduced, thereby improving the reliability of target recognition area localization.

[0011] In some embodiments, after determining the target semantic anchor data, the method further includes: Based on the anchor bounding box size corresponding to the target semantic anchor data, at least one derivative search region is generated according to a preset expansion ratio. Secondary text recognition, color recognition, or control state recognition are performed on the derived search area to obtain supplementary recognition feature data; The supplementary identification feature data is incorporated into the interface feature data.

[0012] In this way, local features can be further extracted around the target area, reducing the impact of full-screen background noise on the recognition results and improving the recognition accuracy of small-sized controls, dense text, or local state areas.

[0013] In some embodiments, when the target semantic anchor data is not obtained, the target interface image data is determined as full-screen recognition image data, and text recognition, template matching, or color recognition are performed on the full-screen recognition image data to obtain the interface feature data.

[0014] Thus, even when the semantic anchor point cannot be determined due to occlusion, language change, recognition failure, or interface abnormality, the interface features can still be extracted through full-screen recognition, avoiding the test process from being directly interrupted due to the failure of a single positioning step.

[0015] In some embodiments, the step of extracting features from the target region image data to obtain interface feature data includes: Multi-scale structural similarity calculation is performed on the target region image data and a preset standard image to obtain structural similarity feature data; The target area image data is compared with the color according to the preset color difference calculation method to obtain color feature data; Local difference detection is performed on the target region image data according to a preset sliding window to obtain local difference feature data; The interface feature data is generated based on at least one of the structural similarity feature data, the color feature data, and the local difference feature data.

[0016] In this way, the interface display status can be described from multiple dimensions such as text, color, overall structure and local anomalies, reducing the problem that a single recognition method is not adaptable to complex interface changes.

[0017] In some embodiments, the first determination method includes: The interface feature data, the expected interface result, and the interface judgment rules are input into a preset interface recognition model to obtain the first interface judgment result; wherein, the first interface judgment result includes at least one of the following: recognition area coordinates, control type, control state, actual text, confidence level, and matching state; The preset confidence condition includes the confidence level being greater than or equal to a preset confidence threshold; The second rule judgment method includes at least one of text matching, color matching, position matching and template matching.

[0018] Thus, when the output of the first judgment method is reliable, it can be used to make a comprehensive judgment on the interface state; when the output of the first judgment method is unreliable, an interpretable supplementary judgment result can be obtained through rule-based comparison, thereby improving the stability of the overall judgment process.

[0019] In some embodiments, when the expected interface result includes a waveform expected result, the method further includes: Extract waveform image data from the target region image data; Edge detection is performed on the waveform image data to obtain waveform point sequence data; Based on the waveform point sequence data, waveform type data and waveform quality parameter data are determined, wherein the waveform quality parameter data includes at least one of overshoot ratio, ringing characteristics, and edge distortion characteristics; The waveform type data and the waveform quality parameter data are compared with the expected waveform result to generate waveform judgment result data; The test conclusion data can be corrected or confirmed based on the waveform judgment results.

[0020] In this way, the waveform display pattern can be detected at the image level, covering display anomalies such as waveform overshoot, ringing, or edge distortion that are difficult to be directly reflected by traditional SCPI return status or numerical return results.

[0021] In some embodiments, when no expected interface feature corresponding to the expected interface result is identified based on the target area image data or full-screen recognition image data, the method further includes: Obtain grayscale statistics of multiple consecutive frames of interface image data; When the grayscale statistics are lower than the preset black screen threshold, black screen abnormal data is generated; When the grayscale statistics are not lower than the preset black screen threshold and the expected interface features are not identified, interface freeze abnormal data is generated. Based on the black screen anomaly data or the interface freeze anomaly data, generate corresponding anomaly test conclusion data.

[0022] In this way, when the expected interface features are not identified, it is possible to further distinguish between black screen, frozen interface, or unknown interface state, which facilitates test report recording and subsequent fault location.

[0023] Secondly, this application provides an interface judgment system after SCPI is issued based on image text recognition, including a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements an interface judgment method after SCPI is issued based on image text recognition.

[0024] The present application will be further described below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the interface judgment method after SCPI issuance based on image text recognition provided in this application embodiment. Detailed Implementation

[0027] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0028] This embodiment provides a method for interface judgment after SCPI (Content Control Point) is issued based on image text recognition, specifically applied to the processor of an electronic device. The electronic device can be a host computer, industrial control computer, server, test terminal, or other computing device with data processing capabilities. The device under test can be an electronic measuring instrument supporting SCPI commands, such as an oscilloscope, signal generator, multimeter, logic analyzer, programmable power supply, etc., or other devices that use similar control commands and have a graphical display interface.

[0029] The processor can send SCPI commands to the device under test (DUT) via Ethernet, USB, serial, GPIB, or other communication interfaces. The processor can acquire interface image data of the DUT through methods such as screenshots, remote display acquisition, video capture, camera photography, or acquisition card capture. This application does not limit the specific method of acquiring interface image data, as long as the interface display image of the DUT after the target SCPI command is obtained.

[0030] like Figure 1 As shown, this embodiment provides a method for determining the interface after SCPI is issued based on image text recognition. The method includes steps S100 to S800.

[0031] Step S100: Obtain test task data, which includes the target SCPI command, expected interface results, and interface judgment rules; send the target SCPI command to the device under test; It is understood that test task data is used to describe a single interface verification task. Test task data can come from test case files, databases, test platforms, or manually configured files. The target SCPI command is used to control the device under test to perform corresponding operations; the expected interface result is used to indicate the display state that the interface of the device under test should present after the target SCPI command is executed; the interface judgment rules are used to limit the judgment conditions used by the processor when determining whether the interface meets expectations.

[0032] For example, the target SCPI command could be a language switching command, a channel activation command, a parameter setting command, a trigger mode setting command, or a menu switching command. Correspondingly, the expected interface result could include the expected displayed text, expected color, expected control state, expected parameter value, expected waveform type, or expected interface structure. Interface judgment rules could include text matching rules, color matching rules, position matching rules, template matching rules, similarity threshold rules, or confidence level rules.

[0033] After acquiring the test task data, the processor sends a target SCPI command to the device under test (DUT). Upon receiving the target SCPI command, the DUT performs the corresponding operation, which may cause changes to its display interface. The processor initiates an interface verification process after the target SCPI command is issued, ensuring that subsequently acquired interface image data corresponds to the interface changes caused by the target SCPI command.

[0034] Before issuing the target SCPI command, the processor can also issue one or more pre-test commands based on the test task data to put the device under test into a preset test state. These pre-test commands may include reset commands, clear status commands, mode setting commands, or basic parameter setting commands. The pre-test commands are used for test preparation, and the interface changes after the target SCPI command is issued are the main focus of interface verification in this embodiment.

[0035] Step S200: After the target SCPI command is sent, the continuous interface image data of the device under test is acquired at a preset time interval, and the pixel difference of the interface image data of two adjacent frames is calculated to obtain the interface change ratio data. After the target SCPI command is issued, the interface of the device under test (DUT) typically requires a certain amount of time to refresh. If a fixed waiting time is used, insufficient waiting time may result in an unstable interface, while excessive waiting time may reduce testing efficiency. Therefore, the processor continuously acquires the interface image data of the DUT at preset time intervals and determines the interface refresh status based on the changes between consecutive interface images.

[0036] The preset time interval can be set according to the type of device under test, interface refresh rate, image acquisition method, or test accuracy requirements. For example, the preset time interval can be 100 milliseconds, 200 milliseconds, 500 milliseconds, or other time intervals. The continuous interface image data includes multiple frames of interface images acquired in chronological order.

[0037] After obtaining continuous interface image data, the processor calculates the pixel difference between two adjacent frames of interface image data. Specifically, it acquires the current frame's interface image data and the previous frame's interface image data, and compares the pixel values ​​at the same coordinate positions in both. When the color channel difference of a pixel at a corresponding position meets a preset difference condition, the processor identifies that pixel as a changed pixel. Subsequently, the processor counts the number of changed pixels and obtains the interface change ratio data based on the ratio of the number of changed pixels to the total number of pixels in the interface image.

[0038] Step S300: Compare the interface change ratio data with a preset stability threshold. When the interface change ratio data is less than the preset stability threshold, determine that the interface of the device under test is in a stable state and acquire the target interface image data. If the interface of the device under test is not determined to be in a stable state within a preset timeout period, generate interface timeout exception data. In this embodiment, the processor determines whether the interface of the device under test has been refreshed based on the interface change ratio data. When the interface change ratio data is less than a preset stability threshold, it indicates that the change between two adjacent frame interface images is small, and the processor can determine that the interface of the device under test is in a stable state and acquire target interface image data for subsequent identification and judgment.

[0039] The preset stability threshold can be configured according to the display characteristics of the device under test. For devices with few interface changes and fast refresh rates, a lower stability threshold can be set; for devices with local dynamic display areas, a higher stability threshold can be set, or the interface change ratio data can be calculated only within a specified area.

[0040] In one implementation, the processor can determine that the interface is in a stable state when the percentage change of the interface data is less than a preset stability threshold. In another implementation, the processor can determine that the interface is in a stable state when the percentage change of the interface data is less than the preset stability threshold multiple times in a row, thereby reducing the possibility of misjudgment caused by abnormal single-frame image acquisition.

[0041] If the processor fails to determine that the interface is in a stable state within a preset timeout period, the processor generates interface timeout exception data. This interface timeout exception data may include the exception type, exception occurrence time, target SCPI command, device under test identifier, last frame of interface image data, and a sequence of interface change percentage data. The interface timeout exception data can be used for subsequent report generation or fault analysis.

[0042] Step S400: Perform text recognition on the target interface image data to obtain interface text data and text position data corresponding to the interface text data. Determine candidate anchor text from the interface text data according to preset semantic anchor data, and perform spatial rationality verification according to the text position data of the candidate anchor text to obtain target semantic anchor data. In this embodiment, the processor performs text recognition on the target interface image data to obtain interface text data and text location data. The interface text data may include menu text, parameter names, unit identifiers, status text, control text, or other interface text. The text location data may include text bounding box coordinates, text center point coordinates, text width, text height, and text recognition confidence score.

[0043] Preset semantic anchor data is used to represent textual information that can serve as a reference for locating the target recognition area. This semantic anchor typically has a relatively stable positional relationship with the target recognition area. For example, when the target recognition area is a parameter value display area, the parameter name to the left or above the parameter value can be selected as the semantic anchor; when the target recognition area is a menu option area, the title of the menu or adjacent fixed text can be selected as the semantic anchor.

[0044] The processor determines candidate anchor text from the interface text data based on preset semantic anchor data. Since there may be multiple identical or similar characters in the interface of the device under test, the processor further performs spatial rationality verification based on the text position data of the candidate anchor text to obtain the target semantic anchor data.

[0045] Step S500: Calculate target recognition region coordinate data based on the target semantic anchor point data, preset normalized offset data, and current interface resolution data; and extract target region image data from the target interface image data based on the target recognition region coordinate data. In this embodiment, after determining the target semantic anchor point data, the processor further determines the target recognition region. The target recognition region is the area in the target interface image data that requires feature extraction and interface judgment.

[0046] Pre-defined normalized offset data is used to represent the relative position of the target recognition region with respect to the target semantic anchor point. This relative position can be predetermined under a standard interface or standard resolution and stored in normalized coordinates. The current interface resolution data is used to represent the width and height of the target interface image data.

[0047] The target semantic anchor point data includes the coordinates of the anchor point's center point and the dimensions of the anchor point's bounding box. The processor calculates the center point coordinates, width, and height of the target recognition region based on the current coordinates of the target semantic anchor point, preset normalized offset data, and the current interface resolution data, thereby obtaining the target recognition region coordinate data. The target recognition region coordinate data may include the coordinates of the top-left corner, bottom-right corner, center point, region width, and region height.

[0048] The processor extracts target region image data from the target interface image data based on the target recognition region coordinate data. This target region image data is used for subsequent feature extraction. If the calculated target recognition region coordinates exceed the range of the target interface image, the processor can perform boundary correction on the target recognition region coordinates or generate region localization anomaly data.

[0049] Step S600: Extract features from the target area image data to obtain interface feature data. The interface feature data includes at least one of text feature data, color feature data, structural similarity feature data, and local difference feature data. In this embodiment, the processor extracts features from the target area image data to obtain interface feature data that characterizes the actual display state of the target area. The interface feature data can be determined based on the expected interface result and interface judgment rules.

[0050] When the expected interface result involves text, the processor can perform text recognition on the target area image data to obtain text feature data. The text feature data can include the actual text within the target area, the text position, the text confidence score, and the text format information. The processor can also normalize the text recognition results, such as removing spaces, standardizing capitalization, standardizing full-width and half-width characters, or standardizing unit formats.

[0051] When the expected interface result involves color, the processor can perform color sampling or color comparison on the target area image data to obtain color feature data. Color feature data may include sampled point color values, area average color values, dominant color values, color histograms, or color difference values.

[0052] When the expected interface result involves the overall structure of the interface, the processor can calculate the structural similarity between the target region image data and a preset standard image to obtain structural similarity feature data. When it is necessary to detect local anomalies, the processor can perform local difference detection on the target region image data to obtain local difference feature data. Local difference feature data may include the coordinates of the difference region, the area of ​​the difference, the type of difference, and the confidence level of the difference.

[0053] Step S700: Based on the interface feature data, the expected interface result, and the interface judgment rule, a first interface judgment result is generated using a first judgment method; when the first interface judgment result does not meet the preset confidence condition, a second judgment method is used to compare the interface feature data with the expected interface result to generate a second interface judgment result. In this embodiment, the processor generates an interface judgment result based on interface feature data, expected interface result, and interface judgment rules. The first judgment method can be a judgment method based on a preset interface recognition model, or a judgment method based on feature fusion scoring or a classifier. The first judgment method is used to judge the interface feature data.

[0054] The preset interface recognition model can be constructed using an object detection network, such as Faster R-CNN or other network structures capable of object region detection and category recognition. The preset interface recognition model can be trained or fine-tuned based on an instrument interface dataset, which may include interface images of different models of the tested device under different functional states, as well as corresponding control area annotations, control type annotations, control status annotations, and text annotations.

[0055] The preset interface recognition model supports the recognition of preset control types in the interface of the device under test. The preset control types include at least one of buttons, drop-down boxes, menus, icons, parameter display areas, status display areas, and waveform areas. The processor can use the target area image data or target interface image data, the interface text data, the text position data corresponding to the interface text data, and the expected interface result as model input to obtain the first interface judgment result.

[0056] In one implementation, the processor inputs interface feature data, expected interface results, and interface judgment rules into a preset interface recognition model to obtain a first interface judgment result. The first interface judgment result may include at least one of the following: recognition area coordinates, control type, control state, actual text, confidence level, and matching state.

[0057] The processor determines whether the judgment result of the first interface meets the preset confidence conditions. The preset confidence conditions may include a confidence level greater than or equal to a preset confidence threshold, a matching status of successful matching, the coordinates of the identified area falling within a preset area range, or a comprehensive score greater than a preset score threshold.

[0058] When the first interface judgment result meets the preset confidence condition, the processor can determine the test conclusion based on the first interface judgment result. When the first interface judgment result does not meet the preset confidence condition, the processor uses the second rule judgment method to compare the interface feature data with the expected interface result and generate the second interface judgment result.

[0059] Step S800: Based on the judgment result of the first interface or the judgment result of the second interface, generate test conclusion data and output test report data including the test conclusion data, the target interface image data, the interface feature data and log data.

[0060] In this embodiment, the processor generates test conclusion data based on the judgment result of the first interface or the judgment result of the second interface. The test conclusion data may include conclusions such as test passed, test failed, interface timed out, recognition anomaly, or pending manual confirmation.

[0061] When the first interface judgment result meets the preset confidence condition, the processor can generate test conclusion data based on the first interface judgment result. When the first interface judgment result does not meet the preset confidence condition and the second interface judgment result has been generated, the processor can generate test conclusion data based on the second interface judgment result. When interface timeout exception data is generated in step S300, the processor can generate exception test conclusion data based on the interface timeout exception data.

[0062] The processor outputs test report data. This test report data may include test conclusion data, target interface image data, interface feature data, and log data. Furthermore, the test report data may also include the target SCPI command, expected interface results, the device under test identifier, target recognition area coordinate data, first interface judgment results, second interface judgment results, anomaly information, screenshot path, judgment time, and judgment process record.

[0063] As can be seen from the above steps S100 to S800, in this embodiment, after the target SCPI command is issued, the interface stability is determined by continuous interface image data. After the interface stabilizes, the target recognition area is determined based on text recognition and semantic anchor points. Feature extraction and layer judgment are performed on the target area image data, so that the actual interface display status of the device under test can be verified in addition to the status returned by the SCPI command.

[0064] In some embodiments, the step of calculating the pixel difference of interface image data of two adjacent frames to obtain interface change ratio data includes: steps S210 to S240.

[0065] Step S210: Obtain the current frame interface image data and the previous frame interface image data; The processor can cache continuously acquired interface image data into an image queue in chronological order. When there are at least two frames of interface images in the image queue, the processor determines the most recently acquired interface image as the current frame of interface image data and the previous frame of interface image data as the previous frame of interface image data.

[0066] Step S220: Compare the pixels at corresponding positions in the current frame interface image data and the previous frame interface image data. The processor compares the pixel values ​​at the same coordinates in two frames of the interface image. Pixel values ​​can be represented using the RGB color space, or they can be represented using grayscale, HSV color space, or other color space values.

[0067] Step S230: When the color channel difference of the pixel at the corresponding position meets the preset difference condition, the pixel is determined as a changed pixel; The processor calculates the differences between the current frame pixel and the previous frame pixel in the R, G, and B channels, respectively. When the difference in at least one color channel is greater than or equal to a preset channel difference threshold, the processor identifies the pixel as a changed pixel.

[0068] In another implementation, the processor can calculate the grayscale difference of the corresponding pixels. When the grayscale difference is greater than or equal to a preset grayscale difference threshold, the pixel is identified as a changed pixel. This method has lower computational complexity and is suitable for interface stability assessment scenarios where color change requirements are not high.

[0069] Step S240: Obtain the interface change ratio data based on the ratio of the number of changed pixels to the total number of pixels in the interface image data.

[0070] The processor counts the number of changed pixels and divides this count by the total number of pixels in the interface image to obtain the interface change ratio data. This interface change ratio data can be expressed as a percentage, a decimal, or other numerical values.

[0071] In some embodiments, the processor may count the number of changed pixels only within a designated stability assessment region, rather than counting the number of changed pixels across the entire interface image. The designated stability assessment region may avoid continuously changing waveform areas, time display areas, or flashing icon areas. This reduces interference from non-target dynamic areas on the interface stability assessment.

[0072] In some embodiments, the step of performing spatial rationality verification based on the text position data of the candidate anchor text to obtain target semantic anchor data includes: steps S410 to S440.

[0073] Step S410: Obtain the current position data and historical anchor position data of the candidate anchor text; The current position data of the candidate anchor text can include the bounding box coordinates, center point coordinates, width, height, and recognition confidence score of the candidate anchor text in the target interface image data. Historical anchor position data can come from the anchor positions saved when the last test was successful, or from the anchor positions pre-annotated in the standard template image.

[0074] Step S420: Calculate displacement data based on the current location data and the historical anchor point location data; Displacement data can include the lateral and vertical offsets of the candidate anchor text center point relative to the historical anchor center points, the magnitude of the displacement vector, and the displacement direction angle. The processor can normalize the displacement data according to the current interface resolution for comparison at different resolutions.

[0075] Step S430: When the displacement data meets the preset position constraint conditions, the corresponding candidate anchor point text is determined as the valid anchor point text; The preset position constraints may include the displacement vector magnitude being less than a preset distance threshold, the displacement direction angle deviation being less than a preset angle threshold, the candidate anchor text being located within a preset anchor search area, the size of the candidate anchor text being within a preset size range, or the text recognition confidence of the candidate anchor text being greater than a preset anchor confidence threshold.

[0076] If the candidate anchor text does not meet the preset position constraints, the processor can identify the candidate anchor text as outlier or interference text and exclude it.

[0077] Step S440: When there are multiple valid anchor texts, determine the target semantic anchor data from the multiple valid anchor texts based on the text recognition confidence score.

[0078] In one implementation, the processor identifies the effective anchor text with the highest character recognition confidence as the target semantic anchor data. In another implementation, the processor calculates a comprehensive score based on character recognition confidence, positional deviation, size matching degree, and directional relationship with the target region, and identifies the effective anchor text with the highest comprehensive score as the target semantic anchor data.

[0079] In some embodiments, after determining the target semantic anchor data, the method further includes: Based on the anchor bounding box size corresponding to the target semantic anchor data, at least one derivative search region is generated according to a preset expansion ratio. The preset expansion ratio can be determined based on the width or height of the anchor point bounding box. For example, the processor can use the anchor point bounding box as a reference and expand its width or height by a preset multiple in the upward, downward, left, and right directions to obtain multiple local search regions.

[0080] Secondary text recognition, color recognition, or control state recognition are performed on the derived search area to obtain supplementary recognition feature data; The supplementary identification feature data may include supplementary text, supplementary color, supplementary control status, supplementary icon information, or supplementary positional relationships in the derived search area.

[0081] The supplementary identification feature data is incorporated into the interface feature data.

[0082] The processor will supplement the identification feature data and incorporate it into the interface feature data, enabling subsequent interface judgments to simultaneously reference features from both the target area image data and the derived search area. This method is suitable for interfaces with small target controls, dense text, or target area boundaries that are difficult to determine precisely.

[0083] In some embodiments, when the target semantic anchor data is not obtained, the target interface image data is determined as full-screen recognition image data, and text recognition, template matching, or color recognition is performed on the full-screen recognition image data to obtain the interface feature data.

[0084] Understandably, when the processor does not obtain the target semantic anchor data, a downgraded recognition process can be executed. That is, in this case, the processor no longer relies on semantic anchors to calculate the target recognition region, but instead searches for interface features corresponding to the expected interface result across the entire screen.

[0085] For example, when the expected interface result is expected text, the processor can perform full-screen text recognition in the full-screen recognition image data and determine whether the expected text exists in the recognition result. When the expected interface result is expected color, the processor can search for areas in the full-screen recognition image data that match the expected color condition. When the expected interface result is expected icon or control structure, the processor can perform template matching on the full-screen recognition image data.

[0086] In some embodiments, the step of extracting features from the target region image data to obtain interface feature data includes: Multi-scale structural similarity calculation is performed on the target region image data and a preset standard image to obtain structural similarity feature data; In structural similarity calculation, the processor can acquire target region image data and a preset standard image. The preset standard image can be a target region image saved under standard test conditions, or a template image generated based on the expected interface result. The processor can perform multi-scale structural similarity calculations on the target region image data and the preset standard image to obtain structural similarity feature data.

[0087] The processor performs multi-scale decomposition on the target region image data and a preset standard image to obtain image data at multiple scales; it calculates the structural similarity between two images at each scale; and then fuses the multiple structural similarities according to the weights corresponding to each scale to obtain structural similarity feature data. Through multi-scale calculation, both the overall structural differences and local structural differences of the target region can be considered simultaneously.

[0088] The target area image data is compared with the color according to the preset color difference calculation method to obtain color feature data; The preset color difference calculation method can be RGB channel difference calculation, HSV color space difference calculation, Lab color space color difference calculation, or other color difference calculation methods. Color feature data can include the average color value, dominant color value, sample point color value, color difference value, or color matching status of the target area.

[0089] Local difference detection is performed on the target region image data according to a preset sliding window to obtain local difference feature data; The size of the preset sliding window can be determined according to the type of the target region. For example, a smaller sliding window can be used for text regions, while a larger sliding window can be used for waveform or graphic regions. The processor can calculate pixel differences, color differences, edge differences, or similarity differences within each sliding window and generate a local difference mask based on the calculation results. The local difference feature data can include the coordinates of the difference region, the difference area, the difference type, and the difference confidence level.

[0090] The interface feature data is generated based on at least one of the structural similarity feature data, the color feature data, and the local difference feature data.

[0091] Furthermore, the processor can adjust the weights of global and local features based on structural similarity feature data. For example, when structural similarity is high, the weight of local difference features can be reduced; when structural similarity is low, the weight of local difference features can be increased to enhance the ability to identify local anomalies.

[0092] In some embodiments, the first determination method includes: The interface feature data, the expected interface result, and the interface judgment rule are input into a preset interface recognition model to obtain the first interface judgment result. The preset interface recognition model can be used to recognize controls, icons, text areas, parameter areas, menu areas, or waveform areas in an interface. The first interface judgment result includes at least one of the following: recognition area coordinates, control type, control state, actual text, confidence level, and matching state. The preset confidence condition includes the confidence level being greater than or equal to a preset confidence threshold; The preset confidence conditions may also include a matching status of "passed," recognition area coordinates falling within a preset area range, and the matching degree between the actual text and the expected text being greater than a preset text matching threshold, or the color matching degree being greater than a preset color matching threshold. When the first interface judgment result meets the preset confidence conditions, the processor can generate test conclusion data based on the first interface judgment result. When the first interface judgment result does not meet the preset confidence conditions, or when the first judgment method executes abnormally, the processor uses a second rule judgment method for supplementary judgment.

[0093] The second rule-based judgment method includes at least one of text matching, color matching, position matching, and template matching. Text matching may include exact matching, containment matching, edit distance matching, or keyword matching. Color matching may include sample point color matching, region average color matching, or color difference threshold matching. Position matching may include whether the coordinates of the identified region fall within a preset region range, or whether the positional relationship between the target control and the semantic anchor point satisfies a preset relationship. Template matching may include whether the similarity between the target region image data and the preset template image is greater than a preset template matching threshold.

[0094] In some embodiments, when the expected interface result includes a waveform expected result, the method further includes: Extract waveform image data from the target region image data; Waveform image data can be the entire image in the target area image data, or it can be a local image extracted based on color, edge, grid area, or preset waveform area coordinates.

[0095] Edge detection is performed on the waveform image data to obtain waveform point sequence data; The processor can first perform grayscale conversion and noise reduction on the waveform image data, and then extract the waveform edge points through edge detection. Subsequently, based on the coordinate relationship of the edge points, the two-dimensional waveform edge is converted into one-dimensional waveform point sequence data.

[0096] Based on the waveform point sequence data, waveform type data and waveform quality parameter data are determined, wherein the waveform quality parameter data includes at least one of overshoot ratio, ringing characteristics, and edge distortion characteristics; Waveform type data can include sine wave, square wave, pulse wave, triangle wave or other waveform types.

[0097] The waveform type data and the waveform quality parameter data are compared with the expected waveform result to generate waveform judgment result data; For example, when the expected waveform result is a square wave and no obvious overshoot is required, the processor can calculate the proportion of the waveform top exceeding the stable level based on the waveform point sequence data to obtain the overshoot ratio; when the overshoot ratio is greater than the preset overshoot threshold, waveform judgment result data of waveform abnormality is generated.

[0098] The test conclusion data can be corrected or confirmed based on the waveform judgment results.

[0099] If the text, color, and other interface features displayed in the first or second interface judgment results are normal, but the waveform judgment result data indicates that the waveform quality parameters do not meet expectations, the processor can determine the test conclusion data as a test failure or waveform abnormality. This embodiment is applicable to test scenarios that require judgment of waveform display morphology.

[0100] In some embodiments, when no expected interface feature corresponding to the expected interface result is identified based on the target area image data or full-screen recognition image data, the method further includes: Obtain grayscale statistics of multiple consecutive frames of interface image data; Grayscale statistics can include average grayscale value, grayscale variance, proportion of low grayscale pixels, or grayscale histogram. The processor can convert color interface images to grayscale images and perform statistical calculations on the grayscale images.

[0101] When the grayscale statistics are lower than the preset black screen threshold, black screen abnormal data is generated; The black screen anomaly data may include an anomaly code, an anomaly description, multiple consecutive frame interface images, grayscale statistics, and corresponding test task information.

[0102] When the grayscale statistics are not lower than the preset black screen threshold and the expected interface features are not identified, interface freeze abnormal data is generated. Interface freeze anomaly data can be used to indicate that the interface of the device under test is not black, but the interface content does not show the expected changes, which may be due to interface freeze, interface jump error or entering an unknown state.

[0103] Based on the black screen anomaly data or the interface freeze anomaly data, generate corresponding anomaly test conclusion data.

[0104] Therefore, when the expected interface features are not identified, the processor can further distinguish between black screen anomalies and non-black screen interface anomalies.

[0105] This embodiment also provides an interface judgment system after SCPI issuance based on image text recognition, including a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the interface judgment method after SCPI issuance based on image text recognition.

[0106] The above description is merely a preferred embodiment of this application and does not constitute any limitation on this application. Any person skilled in the art can make many possible variations and modifications to the technical solution of this application, or modify it into equivalent embodiments, without departing from the scope of the technical solution of this application. Therefore, all equivalent changes made based on the shape, structure, and principle of this application without departing from the content of the technical solution of this application should be covered within the protection scope of this application.

Claims

1. A method for interface judgment after SCPI issuance based on image text recognition, characterized in that, The method includes: Acquire test task data, which includes the target SCPI command, expected interface results, and interface judgment rules; and send the target SCPI command to the device under test. After the target SCPI command is sent, continuous interface image data of the device under test is acquired at preset time intervals, and pixel differences are calculated for interface image data of two adjacent frames to obtain interface change ratio data. The interface change ratio data is compared with a preset stability threshold. When the interface change ratio data is less than the preset stability threshold, the interface of the device under test is determined to be in a stable state, and the target interface image data is acquired. When the interface of the device under test is not determined to be in a stable state within a preset timeout period, interface timeout abnormal data is generated. The target interface image data is subjected to text recognition to obtain interface text data and text position data corresponding to the interface text data. Candidate anchor text is determined from the interface text data according to preset semantic anchor data, and spatial rationality is verified according to the text position data of the candidate anchor text to obtain target semantic anchor data. Based on the target semantic anchor point data, the preset normalized offset data and the current interface resolution data, the target recognition region coordinate data is calculated, and the target region image data is extracted from the target interface image data based on the target recognition region coordinate data. Feature extraction is performed on the target region image data to obtain interface feature data, which includes at least one of text feature data, color feature data, structural similarity feature data, and local difference feature data. Based on the interface feature data, the expected interface result, and the interface judgment rule, a first interface judgment result is generated using a first judgment method; when the first interface judgment result does not meet the preset confidence condition, a second judgment method is used to compare the interface feature data with the expected interface result to generate a second interface judgment result. Based on the judgment result of the first interface or the judgment result of the second interface, test conclusion data is generated, and test report data including the test conclusion data, the target interface image data, the interface feature data, and log data is output.

2. The interface judgment method after SCPI issuance based on image text recognition according to claim 1, characterized in that, The step of calculating the pixel difference between two adjacent frames of interface image data to obtain interface change ratio data includes: Get the current frame's interface image data and the previous frame's interface image data; The pixels at corresponding positions in the current frame interface image data and the previous frame interface image data are compared. When the color channel difference of a pixel at a corresponding position meets a preset difference condition, the pixel is identified as a changed pixel. The interface change ratio data is obtained by comparing the number of changed pixels with the total number of pixels in the interface image data.

3. The interface judgment method after SCPI issuance based on image text recognition according to claim 1, characterized in that, The step of performing spatial rationality verification based on the text position data of the candidate anchor text to obtain target semantic anchor data includes: Obtain the current position data and historical anchor point position data of the candidate anchor point text; Calculate displacement data based on the current location data and the historical anchor point location data; When the displacement data meets the preset position constraint conditions, the corresponding candidate anchor point text is determined as the valid anchor point text; When multiple valid anchor texts exist, the target semantic anchor data is determined from the multiple valid anchor texts based on the text recognition confidence score.

4. The interface judgment method after SCPI issuance based on image text recognition according to claim 3, characterized in that, After determining the target semantic anchor data, the process also includes: Based on the anchor bounding box size corresponding to the target semantic anchor data, at least one derivative search region is generated according to a preset expansion ratio. Secondary text recognition, color recognition, or control state recognition are performed on the derived search area to obtain supplementary recognition feature data; The supplementary identification feature data is incorporated into the interface feature data.

5. The interface judgment method after SCPI issuance based on image text recognition according to claim 1, characterized in that, When the target semantic anchor data is not obtained, the target interface image data is determined as full-screen recognition image data, and text recognition, template matching or color recognition is performed on the full-screen recognition image data to obtain the interface feature data.

6. The interface judgment method after SCPI issuance based on image text recognition according to claim 1, characterized in that, The step of extracting features from the target region image data to obtain interface feature data includes: Multi-scale structural similarity calculation is performed on the target region image data and a preset standard image to obtain structural similarity feature data; The target area image data is compared with the color according to the preset color difference calculation method to obtain color feature data; Local difference detection is performed on the target region image data according to a preset sliding window to obtain local difference feature data; The interface feature data is generated based on at least one of the structural similarity feature data, the color feature data, and the local difference feature data.

7. The interface judgment method after SCPI issuance based on image text recognition according to claim 1, characterized in that, The first determination method includes: The interface feature data, the expected interface result, and the interface judgment rules are input into a preset interface recognition model to obtain the first interface judgment result; wherein, the first interface judgment result includes at least one of the following: recognition area coordinates, control type, control state, actual text, confidence level, and matching state; The preset confidence condition includes the confidence level being greater than or equal to a preset confidence threshold; The second rule judgment method includes at least one of text matching, color matching, position matching and template matching.

8. The interface judgment method after SCPI issuance based on image text recognition according to claim 1, characterized in that, When the expected interface result includes the expected waveform result, the method further includes: Extract waveform image data from the target region image data; Edge detection is performed on the waveform image data to obtain waveform point sequence data; Based on the waveform point sequence data, waveform type data and waveform quality parameter data are determined, wherein the waveform quality parameter data includes at least one of overshoot ratio, ringing characteristics, and edge distortion characteristics; The waveform type data and the waveform quality parameter data are compared with the expected waveform result to generate waveform judgment result data; The test conclusion data can be corrected or confirmed based on the waveform judgment results.

9. The interface judgment method after SCPI issuance based on image text recognition according to claim 1, characterized in that, When no expected interface feature corresponding to the expected interface result is identified based on the target area image data or full-screen recognition image data, the method further includes: Obtain grayscale statistics of multiple consecutive frames of interface image data; When the grayscale statistics are lower than the preset black screen threshold, black screen abnormal data is generated; When the grayscale statistics are not lower than the preset black screen threshold and the expected interface features are not identified, interface freeze abnormal data is generated. Based on the black screen anomaly data or the interface freeze anomaly data, generate corresponding anomaly test conclusion data.

10. A system for determining the interface after SCPI distribution based on image text recognition, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the interface judgment method after SCPI issuance based on image text recognition as described in any one of claims 1 to 9.

Images