A method, device, equipment and medium for automatic testing of in-call seats

Abstract

The present application relates to the field of automation testing, and particularly relates to a method, device, equipment and medium for automatic testing of an inside talk position. Firstly, an interface control tree is traversed, dynamic interference areas are matched according to logical properties of controls, and these areas are subjected to mask processing to obtain a denoising image. Then, a target detection area is located on the image by template matching. Next, state matching in three dimensions of color, position offset and pattern contour is repeatedly performed on the area at fixed time intervals, and continuous matching results are recorded. When the number of successful continuous matching reaches a preset threshold, the visual detection result is matched with a background log keyword matching result and an audio driving state query result for consistency verification, and finally a test conclusion is output. Interference area dynamic positioning is realized by replacing coordinates with control properties, asynchronous response time is adaptively polled, and the accuracy of single state determination is improved by three-dimensional matching.

Description

Technical Field

[0001] This invention relates to the field of automated testing, and more specifically to a method, apparatus, equipment, and medium for automated testing of intercom seats. Background Technology

[0002] With the development of automated testing technology, GUI image recognition-based automated testing methods are gradually being applied to various software testing applications. These general-purpose GUI automated testing tools (such as Xdotool, Selenium, Appium, etc.) simulate user operations by recognizing the coordinates and attributes of interface elements, thus achieving test automation. However, intercom interfaces typically run on specific embedded or industrial control computer systems, whose interface rendering mechanisms differ from ordinary PC software, and which have extremely high real-time requirements.

[0003] The intercom interface contains a large number of non-critical elements that are dynamically refreshed in real time. These dynamically changing elements continuously produce pixel-level changes, and common image comparison algorithms are very likely to misinterpret these normal dynamic changes as interface state changes or recognition failures, leading to frequent false alarms from the script. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device and medium for automated testing of intercom seats, which solves the problems in the prior art.

[0005] This invention is achieved through the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide a method for automated testing of intercom seats, including:

[0007] Based on the logical attribute combination of each control in the control tree of the target software interface, the node-by-node comparison is performed with the pre-configured interference area matching rules. The screen area corresponding to the successfully matched control node is processed by pixel masking to obtain the denoised image to be recognized.

[0008] Obtain a pre-made target control template image, and use a similarity-based matching algorithm on the denoised image to search for the region with the highest similarity to the target control template image to obtain the target detection region;

[0009] At fixed time intervals, state matching detection, which includes three dimensions of color features, position offset and pattern contour, is performed simultaneously on the target detection area to obtain a sequence of multiple consecutive detection results.

[0010] When the number of consecutive successful matches in the detection result sequence reaches the preset stable number threshold, the consistency of the data layer signal obtained by keyword matching from the background log of the target software system and the physical layer signal obtained by querying the underlying audio interface of the target software system is verified to obtain the final test conclusion.

[0011] Preferably, based on the logical attribute combination of each control in the control tree of the target software interface, a node-by-node comparison is performed with the pre-configured interference area matching rules. Pixel masking is then applied to the screen area corresponding to the successfully matched control node to obtain a denoised image to be identified, including:

[0012] The complete control hierarchy and logical attributes of each control in the current interface of the intercom seat software are obtained through the system's underlying auxiliary function interface. An interface control tree model is constructed, and the logical attributes include at least one of the following: control identifier, control name, control class name, and parent control name.

[0013] Load the pre-configured interference zone matching rule file. Each interference zone matching rule consists of at least one combination of matching conditions. Each matching condition includes an attribute name, a matching operator, and a target value. The matching operator can be any one of inclusion matching, exact matching, or regular expression matching.

[0014] For each interference zone matching rule, based on the combination of multiple logical attribute matching conditions in the interference zone matching rule, the interface control tree model is recursively traversed and node-by-node logical attribute comparison is performed. Control nodes that simultaneously satisfy all combinations of conditions in the interference zone matching rule are determined to be interference zone control nodes, and the real-time bounding box coordinates of the interference zone control nodes on the screen are read.

[0015] Capture the original screen image of the current interface of the intercom seat software. Based on the real-time bounding box coordinates, perform a masking operation on all pixels within the range defined by the real-time bounding box coordinates in the original screen image to obtain the denoised image to be recognized.

[0016] Preferably, a pre-made target control template image is obtained, and a similarity-based matching algorithm is used on the denoised image to be recognized to search for the region with the highest similarity to the target control template image, thus obtaining the target detection region, including:

[0017] When the intercom software is in standby mode, a standard image of the target control to be detected is captured as the target control template image. The target control template image contains the complete outline and texture features of the target control.

[0018] The normalized cross-correlation matching method is used to search for the region with the highest similarity to the target control template image on the denoised image to be recognized, and the similarity score is calculated.

[0019] When the similarity score is greater than or equal to the preset matching threshold, the region with the highest similarity score is determined as the target detection region;

[0020] When the similarity score is less than the preset matching threshold, the localization failure process is triggered, the localization failure information is recorded, and the next frame image is obtained to re-localize.

[0021] Preferably, state matching detection, encompassing three dimensions—color features, positional offset, and pattern contour—is performed simultaneously on the target detection region at fixed time intervals to obtain a sequence of consecutive detection results, including:

[0022] The state matching detection of the target detection area is repeatedly performed at a preset sampling time interval.

[0023] The color feature detection method is as follows: extract the RGB values ​​of all pixels in the target control area of ​​the target control template image, calculate the mean and standard deviation of the R channel, G channel and B channel respectively, and set the color threshold range from the mean minus twice the standard deviation to the mean plus twice the standard deviation; extract the RGB values ​​of all pixels in the current target detection area, calculate the proportion of pixels falling within the color threshold range, and determine that the color dimension matching is successful when the pixel proportion is greater than or equal to the preset color proportion threshold.

[0024] The method for detecting position offset is as follows: record the initial position coordinates of the target control template image when it is successfully matched for the first time; in each subsequent polling detection, calculate the pixel offset of the current matching position relative to the initial position coordinates; when the pixel offset is less than or equal to the preset pixel offset threshold, it is determined that the position dimension is successfully matched.

[0025] The pattern contour detection method is as follows: extract the contour of the target control region in the target control template image, calculate multiple Hu moment feature values ​​of the contour, and form a standard shape feature vector; extract the contour of the target control region in the current target detection region and calculate the Hu moment feature values ​​to form the current shape feature vector; calculate the Euclidean distance between the standard shape feature vector and the current shape feature vector, and when the Euclidean distance is less than the preset similarity threshold, the pattern dimension is determined to be successfully matched.

[0026] A single detection result is considered a successful match if the color dimension, position dimension, and pattern dimension are all determined to be a successful match; otherwise, the single detection result is considered a failed match.

[0027] Preferably, the method further includes:

[0028] Set a preset maximum timeout period, starting from the first execution of the status matching detection;

[0029] When the cumulative execution time exceeds the preset maximum timeout time and the number of consecutive successful matches does not reach the stable number threshold, the timeout circuit breaker is triggered, the polling is terminated and the visual status signal is output as false.

[0030] When the number of consecutive successful matches reaches a stable threshold, the output visual state signal is true.

[0031] Preferably, when the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, the consistency of the data layer signal obtained from the target software system's background log through keyword matching and the physical layer signal obtained from the target software system's underlying audio interface is verified to obtain the final test conclusion, including:

[0032] When the number of consecutive successful matches in the detection result sequence reaches the preset stable number threshold, the visual state signal is judged to be true;

[0033] The system monitors the background log stream output of the intercom system in real time. Regular expressions are used to match keywords in each log line. The regular expressions are predefined to correspond one-to-one with the target events of the interface state changes corresponding to the visual state signals. When a target event is matched within a preset monitoring time window, the output data layer signal is true. When the monitoring times out or no match is found in any log, the output data layer signal is false. The target event is key log information in the background log that can identify whether a business operation has been successfully executed or whether its state has changed.

[0034] The system queries the running status of the intercom seat audio driver process and the on / off status of the audio data stream by querying the underlying API of the operating system or the file system interface. When it is confirmed that the audio data stream is in a normal sending or receiving state, the physical layer signal is output as true; otherwise, the physical layer signal is output as false.

[0035] The visual state signal, data layer signal, and physical layer signal are logically consistent. When all three are true, the final test result is "passed"; when all three are false, the final test result is "failed"; when the logical values ​​of the three are inconsistent, it is determined to be an abnormal conflict state.

[0036] Preferably, when an abnormal conflict state is determined, the following operations are performed automatically:

[0037] Save a complete screenshot of the current in-call interface;

[0038] Save a recent snapshot of the intercom seat system's logs;

[0039] Record detailed data of visual state signals, data layer signals, and physical layer signals;

[0040] Generate conflict reports containing timestamps, signal details, and on-site data, and output anomaly conclusions for manual analysis.

[0041] Secondly, embodiments of the present invention provide an automated testing device for intercom seats, comprising:

[0042] The noise reduction module is used to compare the logical attribute combination of each control in the control tree of the target software interface with the pre-configured interference area matching rules node by node, and perform pixel masking on the screen area corresponding to the successfully matched control node to obtain the noise-reduced image to be recognized.

[0043] The matching module is used to acquire a pre-made target control template image, and to search for the region with the highest similarity to the target control template image on the denoised image to be recognized, thereby obtaining the target detection region.

[0044] The detection module is used to simultaneously perform state matching detection on the target detection area at fixed time intervals, including three dimensions: color features, position offset, and pattern contour, to obtain a sequence of multiple consecutive detection results.

[0045] The verification module is used to verify the consistency of data layer signals obtained by keyword matching from the background log of the target software system and physical layer signals obtained by querying the underlying audio interface of the target software system when the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, and to obtain the final test conclusion.

[0046] Thirdly, embodiments of the present invention provide an electronic device, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect described above.

[0047] Fourthly, embodiments of the present invention provide a storage medium storing computer program instructions, which, when executed by a processor, implement the method of the first aspect described above.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] By matching the interference area through the logical attributes of the control tree and performing masking, dynamic pixel changes are eliminated before template matching. Image recognition is no longer affected by non-critical dynamic elements such as signal strength jumps or clock refreshes, and the misjudgment rate caused by dynamic interference is reduced.

[0050] The target detection area is located in real time by template matching. The target position is recalculated after the interface layout changes. The script does not rely on pre-stored fixed coordinates, which improves its adaptability to different resolutions and UI changes.

[0051] The system repeats detection at fixed intervals and only terminates polling when the number of consecutive successful matches reaches a threshold. The actual waiting time changes dynamically with the business response time. There is no need to preset a fixed delay. The script runs stably under communication latency fluctuations and does not generate fixed delay redundancy.

[0052] The single detection integrates the judgment of three dimensions: color, position offset, and pattern outline. The instantaneous anomaly caused by rendering jitter of a single attribute will not cause the matching to fail on its own, thus improving the reliability of single matching.

[0053] After continuous matching meets the standard, the data layer signal of background log matching and the physical layer signal of audio-driven status query are further introduced to verify the consistency of the three parties with the visual signal. The final conclusion is only output when the evidence from multiple sources is consistent. False positives and false negatives caused by interface rendering delay or log writing lag are eliminated, improving the accuracy of the test conclusion. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0055] Figure 1 This is a flowchart illustrating the automated testing method for intercom seats provided by the present invention.

[0056] Figure 2 This is a flowchart illustrating an example of an automated testing method for intercom seats provided by the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0058] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0059] It should be noted that all actions of acquiring signals, information or data in the present invention are performed on the premise of complying with the corresponding data protection laws, regulations and policies in the locality, and under the condition of obtaining authorization granted by the corresponding device owner.

[0060] Embodiment 1

[0061] The existing general GUI automated testing methods have poor application effects on intercom position software, which are specifically manifested in three failure modes.

[0062] The first failure mode is manifested as continuous false alarms in the image recognition stage. The intercom position interface contains a large number of dynamically refreshed non-critical elements such as signal strength value jumps, clock updates at the second level, and continuous fluctuations of volume indicators. These dynamic elements produce pixel-level changes at every moment. The existing image comparison algorithms regard any pixel change in the interface as a state change, and the normal refresh of dynamic elements causes the detection result of the target control to be overwhelmed by the pixel changes in the dynamic interference area. The test script frequently reports interface state changes when no business operation is performed, generating a large number of invalid alarms, so that testers cannot distinguish real state changes from pseudo-changes caused by dynamic refresh from the test results.

[0063] The second failure mode is manifested as insufficient stability in the timing control link. The business operation response time of the intercom position software is not fixed. The voice call setup process involves signaling interaction, network transmission and audio codec, and the sum of the time consumed by each link fluctuates with the network state, so the completion time may be fast or slow. The existing general automated testing methods adopt a fixed delay waiting mode. If the delay is set too short, the subsequent operation is executed before the target state is reached, resulting in step execution failure; if the delay is set too long, the preset full delay length still needs to be waited even when the network is unobstructed, which reduces the execution efficiency of the test script due to delay redundancy, and in test rounds with improperly set delays, the script is directly terminated because the state is not ready.

[0064] The third failure mode is manifested as insufficient reliability of state judgment. The interface rendering relied on by image recognition has rendering delay: when the interface element has been displayed as the target state but the background business logic is not fully ready, the image recognition outputs a success signal but the actual test fails to pass; there is a writing lag in background log writing: when the business has been successfully executed but the log record has not been flushed into the log file, the operation waiting for log matching times out because the target log cannot be read, and the test script determines the actually passed test case as failed.

[0065] The dynamic interference area identification problem, asynchronous response timing problem and multi-source information consistency problem in the above three failure modes affect the operation of the test script from three levels: input image quality, timing control strategy and conclusion judgment basis respectively.

[0066] Therefore, please refer to Figure 1This invention provides a method for automated testing of intercom seats, comprising:

[0067] S1. Based on the logical attribute combination of each control in the control tree of the target software interface, compare it node by node with the pre-configured interference area matching rules, and perform pixel mask processing on the screen area corresponding to the successfully matched control node to obtain the denoised image to be recognized.

[0068] The control tree, obtained through the system's underlying auxiliary function interface, is a data structure reflecting the hierarchical and containment relationships of all control objects in the target software interface. The root node of the control tree is the currently active window, and each child node represents a control object within that window. The connection between parent and child nodes reflects the nesting and containment relationships of controls on the interface. Taking the intercom interface as an example, the root node of the control tree is the intercom main window. The main window contains the channel control area panel, the call control area panel, and the dynamic information area panel. The channel control area panel further contains sub-controls such as the Channel 1 button, the Channel 2 button, and the channel status label.

[0069] Logical properties are static information fields inherent to a control object, used to identify and describe the characteristics of the control. These include the control identifier, control name, control class name, and parent control name. Logical properties do not depend on the control's screen display position or size; instead, they are assigned by the target software when the control is created and remain unchanged. For example, the logical properties of the Channel 1 button include a control identifier of ch1_btn, a control class name of GtkToggleButton, and a parent control name of channel_group.

[0070] Interference zone matching rules refer to a set of rules pre-configured by testers to describe the identification conditions of dynamically refreshed interference zones that need to be blocked. Each rule consists of one or more attribute matching conditions. Each condition specifies an attribute name, a matching operator, and a target value. Matching operators include inclusion matching, exact matching, and regular expression matching. For example, a rule for matching signal strength indicator labels contains three conditions: the class name contains "Label," the name contains "signal," and the parent control name contains "dynamic_info." A rule for matching clock labels contains two conditions: the class name is exactly equal to "GtkLabel," and the name matches "clock" or "time" using a regular expression.

[0071] Pixel masking refers to uniformly replacing all pixels within a specified coordinate range in the original screen image with a preset neutral grayscale pixel value. This prevents the pixels in that area from changing with dynamic interface refreshes, thus eliminating interference from that area in subsequent image recognition processes. Taking the intercom interface as an example, the clock label area updates its displayed value every second, the signal strength indicator area's value changes in real time with the received signal strength, and the volume indicator's progress bar fluctuates continuously with the audio input level. After these areas are identified as interference zones, their corresponding pixel values ​​are all replaced with neutral gray.

[0072] The denoised image to be identified refers to the image data obtained after the original screenshot has been masked. The interference areas that originally had continuous pixel-level dynamic changes in the image have been replaced with fixed pixel values, and no longer produce inter-frame differences.

[0073] Specifically, during the test script execution, each detection loop first captures the original screen image of the current interface. Then, it traverses each control node in the control tree, extracting each of the control's logical attributes and comparing them item by item with the conditions in the interference area matching rules. When all logical attributes of a control satisfy all the conditions defined in a certain matching rule, the control is determined to be an interference area control. After reading the real-time bounding box coordinates of the interference area control from the control tree, a masking operation is performed within the corresponding coordinate range of the original screen image. In the masked image, the pixels of the dynamic interference area have been replaced by neutral grayscale pixels. When this image is used as input for subsequent steps, dynamic refresh of the area will not trigger any pixel changes; therefore, subsequent image matching and state detection only reflect the real state changes of the target detection area.

[0074] In some implementations, S1 involves comparing the logical attribute combinations of each control in the control tree of the target software interface with pre-configured interference area matching rules node by node, and performing pixel masking processing on the screen area corresponding to the successfully matched control node to obtain a denoised image to be identified, including:

[0075] S11. Obtain the complete control hierarchy and logical attributes of each control in the current interface of the intercom software through the system's underlying auxiliary function interface, and construct an interface control tree model. The logical attributes include at least one of the following: control identifier, control name, control class name, and parent control name.

[0076] The system's underlying auxiliary function interface refers to the standardized programming interface provided by the operating system for external programs to access and traverse UI control objects. This interface does not rely on the target software's source code or debugging symbols; instead, it exposes control tree information through the operating system-level auxiliary function framework. On Linux systems, this interface is AT-SPI (Assistive Technology Service Provider Interface), where applications communicate with the control tree service registered with AT-SPI via the D-Bus bus to obtain control information. On Windows systems, this interface is UIAutomation, where applications call system services through the UIAutomation client library to obtain control objects.

[0077] The hierarchy of controls refers to the tree-like structure of containment and inclusion relationships among various control objects in an interface. This relationship reflects the nesting level of controls in the interface layout. For example, a panel control may contain multiple button controls; the panel is the parent control, and the contained buttons are the child controls. After obtaining the complete control hierarchy, the ancestor and descendant nodes of any control can be determined.

[0078] The UI control tree model refers to a structured data object built in the test script's memory based on the acquired control hierarchy and logical attributes. This model retains all nodes and their connections in the original control tree. Each node stores the complete set of logical attributes and the real-time screen bounding box of the corresponding control, providing a data foundation for subsequent interference area matching and traversal.

[0079] Specifically, during the test script initialization phase, a communication connection is established with the target interface control system by calling the system's underlying auxiliary function interface. The interface returns a hierarchical list of all controls in the current intercom window. Each entry contains the control's unique identifier, name string, class name string, parent control identifier, and the control's current screen bounding box coordinates. The test script reads the returned control information item by item and constructs a tree-like data structure in memory according to parent-child relationships. The root node represents the intercom main window, and the test script allocates a memory object for each control node and stores all its logical attributes. The completed control tree model persists in the test script's memory. Each time a test loop needs to locate interference areas, the control attributes are directly read from this model, eliminating the need to repeatedly call the underlying interface.

[0080] S12. Load the pre-configured interference zone matching rule file. Each interference zone matching rule consists of at least one matching condition combination. Each matching condition includes an attribute name, a matching operator, and a target value. The matching operator can be any one of inclusion matching, exact matching, or regular expression matching.

[0081] The interference zone matching rule file is an external configuration file containing all interference zone matching rule definitions, stored in JSON or YAML format. This file is written in a structured data format and is read and parsed when the test script starts. The parsed rule objects are stored as a list in the test script's memory for subsequent matching. Storing rules in a separate configuration file instead of hard-coding them in the script code means that when the interface layout changes and the interference zone identification conditions need to be adjusted, only the configuration file needs to be modified, without recompiling or modifying the test script code.

[0082] The following is an example of a matching rule configuration (YAML format):

[0083] # Configuration file for interference area matching rules: dynamic_area_rules.yaml

[0084] # Each rule consists of multiple matching conditions; a match is successful when all conditions are met simultaneously.

[0085] rules:

[0086] # Rule 1: Match signal strength indicator tag

[0087] - rule_id: "signal_strength_area"

[0088] Description: "Signal strength dynamically refreshes the label"

[0089] match_conditions:

[0090] - property: "ClassName"

[0091] operator: "contains"

[0092] value: "Label"

[0093] - property: "Name"

[0094] operator: "contains"

[0095] value: "signal"

[0096] - property: "ParentName"

[0097] operator: "contains"

[0098] value: "dynamic_info"

[0099] # Rule 2: Match clock tags

[0100] - rule_id: "clock_area"

[0101] Description: "Real-time clock display label"

[0102] match_conditions:

[0103] - property: "ClassName"

[0104] operator: "=="

[0105] value: "GtkLabel"

[0106] - property: "Name"

[0107] operator: "regex"

[0108] value: ".*clock.*|.*time.*"

[0109] The property field specifies the name of the control property to be matched (such as ClassName, Name, ParentName, etc.), the operator field specifies the matching method (contains means contain, == means exact match, regex means regular expression match), and the value field specifies the target value to be matched.

[0110] A matching condition is an independent decision unit in the interference zone matching rules that describes the judgment requirements that a certain logical attribute of a target control should meet. Each matching condition consists of three fields: the attribute name field specifies the type of logical attribute of the control to be matched; the matching operator field specifies the comparison method between the attribute value and the target value; and the target value field specifies the reference value used for comparison. Taking the intercom interface as an example, in the combination of conditions for matching the signal strength label, the first condition has the attribute name as class name, the matching operator as containment matching, and the target value as Label; the second condition has the attribute name as name, the matching operator as containment matching, and the target value as signal; and the third condition has the attribute name as parent control name, the matching operator as containment matching, and the target value as dynamic_info. Only control nodes that simultaneously meet all three conditions are judged as control nodes in the signal strength interference zone.

[0111] Specifically, during the test script startup phase, the configuration file for matching rules in the interference area is read from a specified path. The configuration file is stored in a structured format, and the test script calls the corresponding parser to convert the file content into a list of rule objects in memory. Each rule object contains a rule identifier, rule description text, and a list of matching conditions. Each condition entry in the matching condition list records its attribute name, matching operator type, and target value string. After parsing, the list of rule objects is stored in the test script context for subsequent access during control tree traversal matching.

[0112] S13. For each interference zone matching rule, based on the combination of multiple logical attribute matching conditions in the interference zone matching rule, recursively traverse the interface control tree model and compare the logical attributes of each node. The control nodes that simultaneously satisfy all the condition combinations in the interference zone matching rule are determined as interference zone control nodes, and the real-time bounding box coordinates of the interference zone control nodes on the screen are read.

[0113] Recursive traversal refers to starting from the root node of the control tree, performing a matching check on each node, and if the node has child nodes, continuing to perform the same matching check on each child node until all nodes have been visited. Recursive traversal ensures that each node in the control tree is visited exactly once.

[0114] Node-by-node logical attribute comparison refers to the process of extracting the logical attribute values ​​of each control node encountered during traversal, comparing the extracted attribute values ​​with the matching conditions of the corresponding attribute names in the interference area matching rules, and recording the comparison result of each condition. Only when the comparison results of all matching conditions in the rule are true, does the node satisfy the combination of matching conditions of that rule.

[0115] Real-time bounding box coordinates refer to the actual display position and size of a control node on the screen at the current moment. Bounding box coordinates are in pixels and are typically represented by four values: the top-left x-coordinate, the top-left y-coordinate, the bottom-right x-coordinate, and the bottom-right y-coordinate. These coordinate values ​​are updated in real-time by the control tree model as the interface renders, and are read as the actual position in the current frame.

[0116] Specifically, during each test execution, for each loaded interference zone matching rule, a complete recursive traversal of the control tree model is performed. The traversal starts from the root node. When visiting the root node, the logical attribute values ​​of that node are extracted, and each attribute value is compared with the conditions of the current rule: for each matching condition in the rule, the corresponding attribute value of the control node is extracted and compared with the target value according to the matching operator. Matching requires the attribute value string to contain a substring of the target value; exact matching requires the attribute value string to be exactly the same as the target value; regular expression matching requires the attribute value string to conform to the pattern of the target value's regular expression. When all matching conditions are satisfied, the current control node is determined to be an interference zone control node. Immediately after determination, the real-time bounding box coordinates of the node are read from the control tree model and stored in a list. When all matching conditions of a rule cannot be completely satisfied by any control in the control tree, the rule has no matching result, and the next rule is processed.

[0117] S14. Capture the original screen image of the current interface of the intercom seat software. Based on the real-time bounding box coordinates, perform a masking operation on all pixels within the range defined by the real-time bounding box coordinates in the original screen image to obtain a denoised image to be recognized.

[0118] The raw screen image refers to the complete pixel data of the intercom software window area captured from the current display buffer via the operating system's screen capture interface. This image contains all visual information of the target software interface at the current moment, including pixel changes in dynamically refreshed areas.

[0119] Masking is an image processing operation that replaces pixel values ​​within a specified pixel region of the original screen image. When performing a masking operation, the three color channel values ​​of each pixel within the target region are assigned a preset neutral grayscale pixel value. This operation does not change the image size; it only modifies the pixel data within the target region, while all pixels outside the region remain unchanged.

[0120] Specifically, after collecting the real-time bounding box coordinates of the interference area control nodes, the test script calls the operating system's screen capture interface to capture the original screen image of the current frame of the client area of ​​the intercom software window. The original image is stored in the test script's memory cache in the form of a pixel matrix. Subsequently, for each interference area bounding box coordinate collected in step three, a masking operation is performed on the pixel matrix of the original image: traversing each row and column of pixels from the top left to the bottom right corner within the coordinate range, the red, green, and blue color channel values ​​of each pixel are assigned to preset neutral grayscale pixel values. After all interference area coordinates have been masked, the non-critical areas in the original image that originally had pixel-level dynamic changes have been uniformly replaced with unchanging grayscale areas. The resulting image is passed to subsequent steps as the denoised image to be identified. This image only retains the effective pixel information of the target detection area, and the inter-frame pixel differences of the dynamic interference area have been completely eliminated. Therefore, when using this image as input for template matching, the matching result of the target control will not be affected by the pixel changes of the dynamic interference area.

[0121] S2. Obtain the pre-made target control template image, and use a similarity-based matching algorithm on the denoised image to search for the region with the highest similarity to the target control template image to obtain the target detection region.

[0122] The target control template image refers to a standard reference image captured when the intercom software is in a known standard state, showing the target control to be detected. This image contains the complete outline and texture features of the target control, serving as a matching benchmark for subsequently searching for the target control's location in the denoised image to be recognized. For example, when testing the PTT button function, the target control template image is the standard appearance image of the PTT button when it is not pressed; when testing the call status indicator light, the template image is the standard appearance image of the indicator light when it is off.

[0123] Similarity-based matching algorithms involve sliding a template image window across the image to be searched, calculating the similarity score between corresponding regions of the template image and the image to be searched at each sliding position, and determining the position with the highest similarity score as the best matching position. The similarity score ranges from 0 to 1, with higher values ​​indicating more consistent visual features between the two image regions.

[0124] The target detection region refers to the rectangular region on the denoised image to be recognized that is most similar to the target control template image, which is located by the template matching algorithm. The position of this region is represented by the bounding box coordinates or the center coordinates, which are used to limit the spatial range of subsequent state detection operations.

[0125] Specifically, after acquiring the denoised image to be recognized, a pre-made target control template image stored in the template library is loaded. The template matching algorithm uses the template image as the core, sliding pixel by pixel on the denoised image to be recognized and calculating the normalized cross-correlation value between the template and the covered area at each position. The sliding position corresponding to the highest similarity value is the optimal matching position of the template in the denoised image, and the rectangular area corresponding to this position is determined as the target detection area. The coordinates of the obtained target detection area do not depend on hard-coded pixel coordinates, but are calculated in real time based on the current frame. Therefore, the target detection area can still be accurately located when the interface resolution changes or the window position moves. The position information of this area is passed to subsequent steps as a spatial constraint condition for state matching detection.

[0126] In some implementations, S2 involves acquiring a pre-made target control template image, and using a similarity-based matching algorithm on the denoised image to be identified to search for the region with the highest similarity to the target control template image, thereby obtaining the target detection region, including:

[0127] S21. When the intercom software is in standby mode, capture a standard image of the target control to be detected as the target control template image. The target control template image contains the complete outline and texture features of the target control.

[0128] In standby mode, the intercom software is in a stable state where no service operations are in progress and the interface display is in its default baseline state. In standby mode, target controls are not affected by any operation triggers or state changes, and their appearance presents the most basic and stable visual form. For example, the PTT button is displayed as a default grayed-out, unpressed state in standby mode, and the channel status indicator light is displayed as off.

[0129] The standard image of the target control refers to a copy of the pixel data of the rectangular area containing the target control, captured by screenshotting the screen when the intercom software is in standby mode. This image serves as a reference for subsequent template matching operations, and its quality directly affects the accuracy and stability of the matching and positioning.

[0130] The target control template image refers to the image data object obtained by loading the standard image of the target control after storing it in the template library. This image is pre-created and persistently stored when the intercom software is in standby mode, and is loaded into memory for use by the matching algorithm during each test execution.

[0131] The complete outline refers to the shape information of the outer boundary of the target control in the target control template image. The outline is composed of continuous pixels of the control boundary, reflecting the geometry and size of the control.

[0132] Texture features refer to the visual details of the surface of a target control in a template image, including color distribution, brightness variations, and local pattern patterns. Texture features reflect visual attributes such as the control's fill style, color scheme, and surface markings.

[0133] Specifically, after the test script is first deployed or the target software interface is redesigned, testers launch the intercom software and confirm it is in a standby state with no business operations. Then, a template creation tool is run. This tool captures the current screen of the intercom software window via a screen capture interface. Testers manually select the target control area to be tested on the screenshot, ensuring the selection area is slightly larger than the control's boundary to include its complete outline. The tool crops the pixel data of the selected area into independent image files and saves them to the template library directory. The template images are stored in a lossless compression format to ensure pixel data fidelity, and the file naming convention includes control identifiers and version information for easy management. During subsequent test execution, the test script reads the corresponding template image file from the template library directory and loads it as an in-memory image object, based on the needs of the test cases. This image object retains its original size and pixel data and is used as a matching benchmark throughout the entire test cycle.

[0134] S22. Using the normalized cross-correlation matching method, search for the region with the highest similarity to the target control template image on the denoised image to be recognized, and calculate the similarity score.

[0135] The normalized cross-correlation matching method is an image matching algorithm based on the statistical correlation of pixel grayscale values. This algorithm slides the template image pixel-by-pixel across the search image from left to right and from top to bottom, calculating the normalized cross-correlation coefficient between the corresponding window regions of the template image and the search image at each sliding position. The normalized cross-correlation matching method is adaptive to changes in image brightness and contrast; its calculation results are unaffected by the overall brightness difference between the search image and the template image.

[0136] The similarity score is a numerical value calculated using the normalized cross-correlation matching method. This value represents the degree of visual similarity between a template image and a window region in the image to be searched. The similarity score ranges from 0 to 1, where a score of 1 indicates that the two image regions are completely identical, and a score of 0 indicates that there is no correlation between the two image regions.

[0137] Specifically, after acquiring the denoised image to be recognized and loading the target control template image, the matching algorithm starts at the top-left pixel of the image to be recognized and slides the search window row by row and column by column on the image to be recognized, using the width and height of the template image as the window size. At each pixel position, the algorithm extracts the grayscale values ​​of all pixels within the current search window of the denoised image to be recognized and substitutes them together with the grayscale values ​​of the corresponding pixels in the template image into the normalized cross-correlation calculation formula. During the calculation, the algorithm calculates the mean pixel value of the template image, the mean pixel value of the search window image, the covariance between the template image and the search window, and the variance of each of the template image and the search window, and calculates the normalized cross-correlation coefficient based on these statistics. This coefficient is the similarity score for that search position. After traversing all search positions, the algorithm records the maximum similarity score among all positions and its corresponding search window position.

[0138] S23. When the similarity score is greater than or equal to the preset matching threshold, the region with the highest similarity score is determined as the target detection region.

[0139] The preset matching threshold refers to the lower boundary value of the similarity score required for successful localization. This threshold is a fixed value between 0 and 1. When the highest similarity score reaches or exceeds this threshold, it indicates that there is a region in the image to be identified that is sufficiently similar to the template image, and the localization process is reliable.

[0140] The target detection region refers to the rectangular window area in the denoised image to be recognized that has the highest similarity score to the target control template image and whose score reaches a preset matching threshold. The location information of this region is used to limit the spatial range of subsequent state matching detection. After this region is determined as the target detection region, the subsequent state matching detection operation is only performed within this region, without the need for a global search on the entire denoised image to be recognized.

[0141] Specifically, after calculating all search locations and obtaining the highest similarity score and its corresponding location, the algorithm compares this highest score with a preset matching threshold. When the highest score reaches or exceeds the preset matching threshold, the localization is successful, and the position and size of the search window corresponding to the highest score in the denoised image to be recognized are determined as the target detection region. After determining the target detection region, the test script records the bounding box coordinate information of the region, including the x and y coordinates of the upper left corner of the region, as well as the width and height of the region. This coordinate information is continuously used as the spatial constraint range for state matching detection in subsequent polling detection processes.

[0142] S24. When the similarity score is less than the preset matching threshold, the localization failure process is triggered, the localization failure information is recorded, and the next frame image is obtained to re-localize.

[0143] The next frame image refers to the new, denoised image to be identified, which is captured again by the test script in the next detection loop after the current frame localization fails. The purpose of using a new frame image instead of reusing the current frame image during localization is to obtain the rendering state of the interface at subsequent moments and avoid single-frame localization failures caused by instantaneous rendering anomalies during interface switching or before the target control has been fully drawn.

[0144] Specifically, when the calculated highest similarity score is lower than the preset matching threshold, the localization process is deemed a failure. The test script then generates a localization failure log containing the current timestamp, the target control identifier, and the reason for the failure. After completing the log, the test script does not proceed to the subsequent state matching detection steps but returns to step one and waits for the next detection cycle. In the next detection cycle, the test script re-captures the original screen image, performs control tree traversal matching, generates a new frame of denoised image to be identified after mask processing, and then executes the template matching localization process again. By re-capturing the current frame image for localization, the test script can automatically recover from localization failures caused by short-term interface anomalies, avoiding the termination of the entire test process due to a single frame localization failure.

[0145] S3. At fixed time intervals, perform state matching detection on the target detection area simultaneously, including three dimensions: color features, position offset, and pattern contour, to obtain a sequence of multiple consecutive detection results.

[0146] The fixed time interval refers to the time length between each state matching detection. This time length is a preset fixed value used to control the frequency of polling sampling.

[0147] Color features refer to the numerical distribution characteristics of all pixels within the target detection area in the RGB color space. Color features establish a reference range by statistically analyzing the mean and standard deviation of each color channel in the target control area of ​​the template image, used to determine whether the color of the current target detection area matches the template. For example, when detecting whether a PTT button is in a call state, if the button changes from the default gray to blue, the RGB distribution of the current area will shift relative to the template image, and the color dimension will determine whether a state change has occurred based on this.

[0148] Position offset refers to the Euclidean distance offset in the screen coordinate system between the matching position of the target detection area in the current frame and its initial position when a successful match is first achieved. It is used to determine whether the target control has undergone spatial displacement beyond the allowable range. Taking the intercom interface as an example, minor jitter or rendering delays in the interface may cause the target detection area to drift by several pixels between consecutive frames. Position offset detection is used to eliminate the interference of this drift on the state determination.

[0149] Pattern contours refer to the geometric shape features of target controls within a target detection region. Pattern contours are described by calculating the Hu moments feature vector of the image region. Hu moments are invariant to translation, rotation, and scaling, and are used to determine whether the shape of the current target region matches the template. For example, when a PTT button changes from a released state to a pressed state, the button's geometric contour changes, and its Hu moment feature vector changes accordingly. Pattern contour detection can recognize this shape change.

[0150] The detection result sequence refers to the data set formed by arranging the successful or unsuccessful matching results of each detection in chronological order after performing state matching detection sequentially at fixed time intervals.

[0151] Specifically, in each round of sampling, the target detection area is simultaneously detected in three dimensions: color, position, and pattern. For color, the RGB values ​​of each pixel in the current area are extracted, and the proportion of pixels falling within a preset color threshold is calculated. A match is determined when this proportion meets a preset requirement. For position, the pixel offset of the current matching position relative to the initial position is calculated. A match is determined when this offset does not exceed a preset offset threshold. For pattern, the contour Hu moment feature vector of the current area is extracted, and the Euclidean distance between it and the template feature vector is calculated. A match is determined when this distance is less than a preset threshold. Only when all three dimensions are determined to be a match is the sampling result recorded as a successful match; otherwise, it is recorded as a failed match. After each sampling, a preset fixed time interval is waited before the next sampling. The matching results generated by the sampling are added to the detection result sequence in chronological order. Since the three dimensions determine the target control from three different perspectives—color attribute, spatial position attribute, and geometric shape attribute—a single successful match indicates that the target control is consistent with the template in all three attribute dimensions.

[0152] In some implementations, S3 involves simultaneously performing state matching detection on the target detection area at fixed time intervals, encompassing three dimensions: color features, positional offset, and pattern contour, to obtain a sequence of consecutive detection results, including:

[0153] S31. Repeatedly perform state matching detection on the target detection area at a preset sampling time interval;

[0154] The preset sampling time interval refers to the time length between two consecutive state matching detections. This time interval is one of the test script configuration parameters, preset before test execution, and remains unchanged throughout the polling process. Taking the internal call establishment test as an example, when the preset sampling time interval is 200 milliseconds, the script performs a state matching detection on the target detection area every 200 milliseconds.

[0155] Specifically, after successfully locating the target detection area, the test script enters a polling sampling loop. Before each sampling execution, the script records the current system timestamp. After completing a state matching detection, the script calculates the time taken for this detection. If the time taken is less than the preset sampling time interval, the script enters a waiting state until the accumulated time from the start of this detection reaches the preset sampling time interval; if the time taken has reached or exceeded the preset sampling time interval, the next sampling begins immediately. Through this waiting mechanism, the time interval between the start times of two adjacent samplings remains at the preset sampling time interval, ensuring that the sampling frequency remains stable throughout the polling process. The matching results generated by each sampling are sequentially appended to the end of the detection result sequence.

[0156] S32. The color feature detection method is as follows: extract the RGB values ​​of all pixels in the target control area of ​​the target control template image, calculate the mean and standard deviation of the R channel, G channel and B channel respectively, and set the color threshold range from the mean minus twice the standard deviation to the mean plus twice the standard deviation; extract the RGB values ​​of all pixels in the current target detection area, calculate the proportion of pixels falling within the color threshold range, and determine that the color dimension matching is successful when the pixel proportion is greater than or equal to the preset color proportion threshold.

[0157] In this context, RGB values ​​refer to the brightness values ​​of each pixel in the red, green, and blue color channels of an image. In image representation, each channel has a value ranging from 0 to 255, and the combination of the three channels determines the final displayed color of the pixel.

[0158] The mean is the arithmetic average of all pixels within an image region on a specific color channel. For the R channel in the target control area of ​​the template image, the sum of the R values ​​of all pixels divided by the total number of pixels gives the R channel mean. The mean of each of the three channels is calculated independently, resulting in the R channel mean, G channel mean, and B channel mean.

[0159] Standard deviation refers to the dispersion of pixel values ​​in a given color channel within an image region relative to the channel's mean. A larger standard deviation indicates a more dispersed distribution of pixel values ​​in that channel, while a smaller standard deviation indicates a more concentrated distribution of pixel values.

[0160] The color threshold range refers to the acceptable range established based on the pixel color statistical features of the target control area in the template image to determine whether the color of the current target detection area is consistent with the template. The color threshold range is set independently for the R, G, and B channels. The lower bound of the threshold range for each channel is the mean of that channel minus twice the standard deviation, and the upper bound is the mean of that channel plus twice the standard deviation.

[0161] The preset color proportion threshold refers to the minimum percentage of pixels in the current target detection area that must fall within the color threshold range to be considered a successful color dimension match. This threshold is a pre-set configuration parameter with a value between 0 and 1. Taking the PTT button in the intercom seat as an example, the button in the template image is gray, with a mean R channel value of 180 and a standard deviation of 10, and a color threshold range of 160 to 200. When the PTT button turns blue, the overall R channel pixel value in the current area drops to about 100, and the R value of a large number of pixels is lower than 160. The percentage of pixels falling within the color threshold range is lower than the preset color proportion threshold, and the color dimension is determined to be a mismatch.

[0162] Specifically, after the initial successful matching and localization, the test script processes the target control template image: extracting all pixel data of the target control region in the template image, and calculating the mean and standard deviation of each of the R, G, and B channels. For each channel, the mean is calculated according to the formula. :

[0163] ;

[0164] Calculate the standard deviation using the formula. :

[0165] ;

[0166] Where the subscript 'c' represents the R, G, or B channel, 'N' represents the total number of pixels, and 'p' represents the pixel value. After obtaining three sets of means and standard deviations, the color threshold range for each channel is set as follows: .

[0167] Specifically, the aforementioned threshold range covers approximately 95% of the template pixel distribution. In each subsequent polling detection, the test script extracts all pixels in the current target detection region. For each pixel, it checks whether its R value falls within the R channel threshold range, its G value falls within the G channel threshold range, and its B value falls within the B channel threshold range. Only pixels falling within all three channels are counted as valid pixels. The proportion of the total number of valid pixels to the total number of pixels in the target detection region is calculated and compared with a preset color proportion threshold. If the color proportion threshold is reached or exceeded, the color dimension is considered a successful match.

[0168] S33. The method for detecting position offset is as follows: record the initial position coordinates of the target control template image when it is successfully matched for the first time; in each subsequent polling detection, calculate the pixel offset of the current matching position relative to the initial position coordinates; when the pixel offset is less than or equal to the preset pixel offset threshold, it is determined that the position dimension is successfully matched.

[0169] The initial position coordinates refer to the x and y coordinates of the top-left corner of the target detection area when the target control template image first meets the preset matching threshold on the denoised image to be recognized. These coordinates are fixed and recorded after the first successful localization and are not updated in subsequent polling processes, serving as the reference origin for position offset calculation.

[0170] Pixel offset refers to the spatial distance between the top-left corner coordinates of the target detection area obtained by template matching and localization in the current polling detection and the initial position coordinates. This distance is expressed in pixels. When the offset is small, it indicates that the position of the target control on the interface remains relatively stable; when the offset is large, it indicates that the target control has undergone significant positional movement.

[0171] The preset pixel offset threshold refers to the maximum pixel offset allowed for a successful positional dimension match. This threshold is a pre-set configuration parameter used to define the normal fluctuation range of the target control's position. Taking the intercom interface as an example, slight interface jitter may cause the target detection area to drift by several pixels between adjacent frames. When the preset pixel offset threshold is set to 5 pixels, positional changes within 5 pixels of offset are considered normal interface stability.

[0172] Upon successful initial matching and localization, the test script records the coordinates of the top-left corner of the target detection area as the initial position coordinates. In subsequent polling detections, the test script re-executes template matching within the current target detection area to obtain the position coordinates of the target control in the current frame. After calculating the difference between the current coordinates and the initial position coordinates in both the horizontal and vertical directions, it calculates the pixel offset using the Euclidean distance formula. The calculated pixel offset is compared with a preset pixel offset threshold. If the offset does not exceed the threshold, the position dimension is considered a successful match; if the offset exceeds the threshold, the position dimension is considered a failed match.

[0173] S34. The detection method for pattern contours is as follows: extract the contour of the target control region in the target control template image, calculate multiple Hu moment feature values ​​of the contour, and form a standard shape feature vector; extract the contour of the target control region in the current target detection region and calculate the Hu moment feature values ​​to form the current shape feature vector; calculate the Euclidean distance between the standard shape feature vector and the current shape feature vector, and when the Euclidean distance is less than the preset similarity threshold, the pattern dimension is determined to be successfully matched.

[0174] Among them, Hu moment eigenvalues ​​refer to a set of numerical features with translation invariance, rotation invariance, and scaling invariance calculated based on the geometric moments of the image region. The Hu moment consists of seven eigenvalues, each describing a different geometric attribute of the image region. The combination of all eigenvalues ​​can uniquely characterize the shape features of the target region.

[0175] A standard shape feature vector is a numerical vector formed by arranging multiple Hu moment feature values ​​of the target control area in a fixed order in the target control template image.

[0176] The current shape feature vector is a numerical vector formed by arranging multiple Hu moment feature values ​​of the target control region in the current target detection area in the same order as the standard shape feature vector.

[0177] Specifically, during the test script initialization phase, contour extraction is performed on the target control template image: after edge detection of the template image, the outer contour curve of the target control is extracted. Based on the extracted contour, seven Hu moment feature values ​​are calculated and arranged sequentially to form a standard shape feature vector. This vector is pre-calculated once and stored in memory after the template is loaded, and then directly reused in subsequent polling detections. In each polling detection, the same contour extraction operation is performed on the current target detection region. After obtaining the contour curve of the current target control region, its seven Hu moment feature values ​​are calculated and arranged to form the current shape feature vector. ;

[0178] Calculate the Euclidean distance between the standard shape feature vector and the current shape feature vector. :

[0179] ,

[0180] The Euclidean distance is compared with a preset similarity threshold. Because Hu moments are translation-invariant, rotation-invariant, and scaling-invariant, this detection method can still accurately identify the consistency of the shape features with the template even when the target control undergoes geometric transformations due to window movement, resolution changes, or interface scaling. When the Euclidean distance is less than the preset similarity threshold, it indicates that the shape of the current target detection region is geometrically consistent with the shape of the template image, and the pattern dimension is determined to be a successful match.

[0181] S35. When the color dimension, position dimension, and pattern dimension are all determined to be a successful match, the single detection result is a successful match; otherwise, the single detection result is a failed match.

[0182] Specifically, in each round of sampling, the test script sequentially executes color feature detection, position offset detection, and pattern contour detection, with each dimension generating its own independent judgment result. The test script performs a logical AND operation on the judgment results of the three dimensions. A successful match is declared when all three conditions—color, position, and pattern—are met simultaneously. A failed match is declared when any one of the three dimensions fails to match. Each detection result is appended to the end of the detection result sequence as input data for subsequent stable counting determinations.

[0183] In some implementations, the method further includes:

[0184] Set a preset maximum timeout period, starting from the first execution of the status matching detection;

[0185] When the cumulative execution time exceeds the preset maximum timeout time and the number of consecutive successful matches does not reach the stable number threshold, the timeout circuit breaker is triggered, the polling is terminated and the visual status signal is output as false.

[0186] When the number of consecutive successful matches reaches a stable threshold, the output visual state signal is true.

[0187] The preset maximum timeout refers to the maximum duration for which the test script is allowed to continuously poll, starting from the first execution of the state matching detection. This timeout is a pre-set configuration parameter, and its value is determined based on the expected completion time limit of the target business operation. Taking the establishment of an intercom voice call as an example, the voice communication establishment process involves multiple stages such as signaling interaction, network transmission, and audio encoding and decoding. Under normal network conditions, it can be completed within 1 second, while under network congestion, it may take several seconds. Setting the preset maximum timeout to 10 seconds covers the completion time range under both normal and congested network conditions.

[0188] Circuit breaking due to timeout is a protection mechanism that forcibly terminates the test script from continuing polling when the cumulative execution time of polling exceeds the preset maximum timeout period and the target state has not yet been achieved. This mechanism prevents the test script from endlessly waiting when the target state cannot be achieved due to abnormal conditions, ensuring that the test process can end within a defined time and return a clear conclusion.

[0189] The visual state signal refers to the logic value signal output after the stable counting judgment or timeout circuit breaker in step three. This signal represents the interface state detection conclusion obtained based on image recognition and state matching. A true visual state signal indicates that the target state has been successfully confirmed through multiple consecutive matches, while a false visual state signal indicates that the target state was not achieved within the preset maximum timeout period.

[0190] Specifically, after successful target detection and before the first execution of state matching detection, the test script records the current system timestamp as the polling start time. After each state matching detection is completed, the test script calculates the difference between the current system time and the polling start time to obtain the cumulative execution time. When the number of consecutive successful matches reaches a stable threshold, it indicates that the target state has been confirmed in multiple consecutive samples, and a single successful match is no longer a coincidental pixel occurrence, but rather a confirmation that the target control state has indeed changed and remained stable. The test script immediately exits the polling loop and outputs a true visual state signal. When the cumulative execution time exceeds the preset maximum timeout and the number of consecutive successful matches has not yet reached the stable threshold, it indicates that the target state has not been achieved within the allowed time limit. The test script immediately exits the polling loop and outputs a false visual state signal. Of the two termination conditions, the first condition met determines when the polling terminates. When the target state reaches the stable confirmation condition at any time within the preset maximum timeout period, the polling terminates immediately, with the waiting time exactly equal to the actual time the target state is achieved. When the target state fails to be achieved within the preset maximum timeout period, the polling terminates when the maximum timeout period is reached, with the waiting time equal to the preset maximum timeout period. This mechanism allows the actual waiting time of the test script to adapt to the completion time of the target business operation, avoiding the situation where, under a fixed delay waiting method, the subsequent steps begin before the state is achieved due to an excessively short delay setting, or the situation where, due to an excessively long delay setting, each round of testing consumes a fixed length of redundant waiting time.

[0191] S4. When the number of consecutive successful matches in the detection result sequence reaches the preset stable number threshold, the consistency of the data layer signal obtained by keyword matching from the background log of the target software system and the physical layer signal obtained by querying the underlying audio interface of the target software system is verified to obtain the final test conclusion.

[0192] The stability threshold refers to the pre-set number of consecutive successful matches required to determine whether the target state has been stably achieved. Only when the number of consecutive successful matches accumulates to this threshold is the target state considered confirmed from the momentary pixel coincidence, and the state is considered validly achieved. For example, if the preset stability threshold is 3, three consecutive samples must be determined to be successful before the target state is confirmed to have been stably achieved.

[0193] Data layer signals refer to logical values ​​obtained from the target software system's background log stream through regular expression keyword matching. These signals indicate whether a target event record related to the business operation corresponding to the target detection area has appeared in the background log. Taking an internal call scenario as an example, when step three detects the PTT button turning blue, the target event corresponding to the data layer signal is a log line in the background log containing "Call Established" and "Type: Radio," matched using the regular expression "Call Established.*Type: Radio."

[0194] Physical layer signals are logical values ​​obtained by querying the running status and audio data flow status of the intercom seat audio driver process through the operating system's low-level interface or file system interface. These signals characterize whether the audio data stream is in a normal sending or receiving state. For example, in a Linux system, the status information of the audio driver process can be queried through the / proc file system to determine the connectivity of the audio data stream.

[0195] Consistency verification refers to the process of comparing the logical values ​​of multiple signals from different data sources to determine whether the logical values ​​of each signal are consistent, and making different processing decisions based on whether they are consistent.

[0196] Specifically, once the cumulative number of consecutive successful matches in the detection result sequence reaches a preset stable threshold, the states of three independent signal sources are acquired. The visual state signal is the state result used in step three to determine the consecutive successful matches. The data layer signal is obtained by real-time monitoring of the background log stream within a preset listening time window and performing regular expression matching; if the target event is matched within the time window, it is true; otherwise, it is false. The physical layer signal is obtained by querying the audio driver state interface at the operating system's underlying level; if the audio data stream is in a normal sending or receiving state, it is true; otherwise, it is false. Logical values ​​of the three signals are compared. When all three are true, the evidence from the visual, log, and audio dimensions corroborates each other, and a pass conclusion is output. When all three are false, the evidence from the three dimensions corroborates each other, indicating a failed test, and a fail conclusion is output. When the logical values ​​of the three are not completely consistent, it indicates an information contradiction between the visual, data, and physical layers, and an abnormal conflict conclusion is output. Based on verification from three independent data sources, the delay or anomaly of any single data source will not independently determine the conclusion; a definitive conclusion is only output when the evidence from multiple sources is consistent.

[0197] In some embodiments, 6. the method according to claim 1, characterized in that, when the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, the data layer signal obtained from the target software system background log through keyword matching and the physical layer signal obtained from the target software system's underlying audio interface are used for consistency verification to obtain the final test conclusion, including:

[0198] When the number of consecutive successful matches in the detection result sequence reaches the preset stable number threshold, the visual state signal is judged to be true;

[0199] The system monitors the background log stream output of the intercom system in real time. Regular expressions are used to match keywords in each log line. The regular expressions are predefined to correspond one-to-one with the target events of the interface state changes corresponding to the visual state signals. When a target event is matched within a preset monitoring time window, the output data layer signal is true. When the monitoring times out or no match is found in any log, the output data layer signal is false. The target event is key log information in the background log that can identify whether a business operation has been successfully executed or whether its state has changed.

[0200] The system queries the running status of the intercom seat audio driver process and the on / off status of the audio data stream by querying the underlying API of the operating system or the file system interface. When it is confirmed that the audio data stream is in a normal sending or receiving state, the physical layer signal is output as true; otherwise, the physical layer signal is output as false.

[0201] The visual state signal, data layer signal, and physical layer signal are logically consistent. When all three are true, the final test result is "passed"; when all three are false, the final test result is "failed"; when the logical values ​​of the three are inconsistent, it is determined to be an abnormal conflict state.

[0202] The background log stream refers to the continuous text-based sequence of operational records output by the intercom software during its operation. Each record in the log stream includes a timestamp, log level, and event description. Every business operation performed by the software and every state change will generate a corresponding record entry in the log stream.

[0203] Regular expressions are sequences of strings used to describe text matching patterns. These sequences define matching conditions for character combinations according to regular expression syntax rules. Regular expressions can perform pattern matching operations on log lines, checking whether the log text contains substrings that match a predetermined pattern. Taking an intercom wireless call scenario as an example, when a log line containing a successful wireless call setup message appears in the background log, the regular expression `Call Established.*Type: Radio` can match the string "Call Established" in the log line and capture the subsequent "Type" field information, thus confirming that the log line corresponds to a wireless call setup event.

[0204] The preset listening time window refers to the maximum time from when the visual state signal becomes true, allowing the test script to listen to the background logs and wait for the target event to occur. This time length is the same as the preset maximum timeout, ensuring that visual layer detection and data layer detection are completed within the same time scale.

[0205] A target event refers to key log information in the background log that identifies the successful execution of a specific business operation or a change in a specific state. The definition of a target event corresponds one-to-one with the interface state change corresponding to a visual state signal; that is, each expected visual state change in the interface has a corresponding background log event as its data layer evidence. Taking an intercom call test as an example, when step three detects that the PTT button turns blue, the corresponding interface state change is that the PTT button is pressed and the call is initiated. The target event corresponding to this state change is the appearance of a log line in the background log indicating that a wireless call has been established.

[0206] Operating system low-level APIs refer to the programming interfaces provided by the operating system kernel for external programs to query system running status and process information. Through this interface, the current status of the audio driver process can be directly obtained, including whether the process is alive, its resource usage, and the audio stream processing status.

[0207] The file system interface refers to the kernel status information access path exposed through the operating system's virtual file system. In Linux systems, the / proc file system provides process status information in the form of files. By reading the status file corresponding to the audio driver process in the / proc file system, one can obtain whether the process is currently active and its audio stream processing count.

[0208] An audio data stream refers to a continuous data channel carrying audio signals in an intercom system. The audio data stream includes two directions: uplink and downlink. Uplink refers to audio data collected from the local microphone being sent to a remote location, while downlink refers to audio data received from a remote location being played back to the local speaker.

[0209] Specifically, by monitoring the background log stream in real time during test case execution and initiating target event matching after the visual state signal is triggered, as well as querying the audio driver process status and audio data flow interruption status, the test script obtains three independent pieces of evidence from the interface layer, data layer, and physical layer. When performing a logical consistency check on the three signals, if all three are true, the evidence from the three dimensions—visual interface change, background log recording, and audio physical continuity—matches each other, resulting in a pass conclusion. If all three are false, the evidence from all three dimensions points to the business operation not being executed, resulting in a fail conclusion. If the logical values ​​of the three are inconsistent—for example, the visual state signal is true but the data layer signal is false, or the visual state signal is true but the physical layer signal is false—it indicates an information contradiction between interface rendering, log recording, and audio driver, resulting in an abnormal conflict conclusion. After verification based on three independent data sources, any transient anomalies caused by rendering delays, log write lags, or driver query delays in any data source will not individually determine the final conclusion; a definitive conclusion will only be output when the evidence from multiple sources is consistent.

[0210] In some implementations, when an abnormal conflict state is determined, the following operations are automatically performed:

[0211] Save a complete screenshot of the current in-call interface;

[0212] Save a recent snapshot of the intercom seat system's logs;

[0213] Record detailed data of visual state signals, data layer signals, and physical layer signals;

[0214] Generate conflict reports containing timestamps, signal details, and on-site data, and output anomaly conclusions for manual analysis.

[0215] Specifically, upon determining an abnormal conflict state, the test script immediately performs a live data saving operation. When saving a complete screenshot, the script calls the screen capture interface to obtain the pixel data of the current interface and writes the image to the storage directory. The image file name includes the test case identifier and the current timestamp. When saving a log snapshot, the script calculates the start time between the anomaly determination time and the preset backtracking duration, extracts log entries within that time range from the system log file, and writes them to the storage file. When recording detailed data, the script reads the raw score data of the three-dimensional matching, the matching results and matching time of the regular expression, and the return status and query time of the audio-driven query from the test context variables. It then organizes this data into structured data blocks according to the signal source and persists them. After completing the above data saving, the script generates a conflict report. The report header includes the timestamp of the anomaly determination, the main body is filled with the logical values ​​of the three signals and their corresponding detailed data, and the end of the report includes the storage path of the complete screenshot and the file path of the log snapshot. The conflict report is output in a structured document format to the test results directory for testers to review and analyze the root causes of the anomaly. This output is an abnormal conclusion, which is different from the two definitive conclusions of pass and fail. The abnormal conclusion does not make a qualitative judgment on the test result as pass or fail, but rather indicates that there is a signal conflict situation that requires manual intervention and analysis by the tester, who should make a comprehensive judgment based on the on-site data.

[0216] For example, such as Figure 2 As shown, the wireless communication test based on traversing the shielded interference area using a control tree follows these specific steps:

[0217] Step 1: Shielding of dynamic interference areas:

[0218] Step 1.1: After the test script starts, it obtains the complete control tree of the intercom seat window through the system's underlying auxiliary function interface;

[0219] Step 1.2: Load the interference region matching rules from the configuration file;

[0220] Step 1.3: Traverse the control tree and match the current bounding box coordinates of the control in the interference area according to the rules;

[0221] Step 1.4: After capturing the original image of the current screen, set all pixels within the coordinate range of the interference area to gray (mask).

[0222] Step 1.5: Output the denoised image to be recognized and cache it in memory for use in Step 2.

[0223] Step 2: Multi-dimensional state polling detection:

[0224] Step 2.1: Set the timeout Tmax = 10s, the sampling interval Δt = 200ms, and the number of stabilization cycles N = 3;

[0225] Step 2.2: Extract the area containing the PTT button from the denoised image to be recognized;

[0226] Step 2.3: Perform multi-dimensional matching on this region every 200ms:

[0227] A) Color dimension: Detect whether the RGB values ​​fall within the blue range;

[0228] B) Position dimension: Calculate the centroid of the button icon outline and check if the offset is ≤5 pixels;

[0229] C) Calculate the shape Hu moment and compare it with the template library to see if the similarity is ≥85%.

[0230] Step 2.4: If all three dimensions match successfully for three consecutive times, immediately return V=True;

[0231] Step 2.5: If the condition is not met even after Tmax is reached, the circuit breaker is triggered, and V=False is returned immediately.

[0232] Step 3: Triple closed-loop verification:

[0233] Step 3.1: Visual signal V, directly obtain the return value from step 2;

[0234] Step 3.2: Log signal L, monitor the background logs in real time, and match the chain establishment success event using regular expressions;

[0235] Step 3.3: For audio signal A, query the running status and call count of the audio driver process through the / proc file system or system API;

[0236] Step 3.4: Comprehensive Judgment:

[0237] A) If V=True, L matches, and A is normal, output "PASS";

[0238] B) If V=False, L does not match, and A is abnormal, output "FAIL";

[0239] C) If V, L, and A are inconsistent (e.g., V=True but A is abnormal), output "ABNORMAL" and save the screenshot and log snapshot.

[0240] Example 2

[0241] This invention provides an automated testing device for intercom seats, comprising:

[0242] The noise reduction module is used to compare the logical attribute combination of each control in the control tree of the target software interface with the pre-configured interference area matching rules node by node, and perform pixel masking on the screen area corresponding to the successfully matched control node to obtain the noise-reduced image to be recognized.

[0243] The matching module is used to acquire a pre-made target control template image, and to search for the region with the highest similarity to the target control template image on the denoised image to be recognized, thereby obtaining the target detection region.

[0244] The detection module is used to simultaneously perform state matching detection on the target detection area at fixed time intervals, including three dimensions: color features, position offset, and pattern contour, to obtain a sequence of multiple consecutive detection results.

[0245] The verification module is used to verify the consistency of data layer signals obtained by keyword matching from the background log of the target software system and physical layer signals obtained by querying the underlying audio interface of the target software system when the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, and to obtain the final test conclusion.

[0246] It should be noted that each module and unit in the automated testing device for intercom seats in this embodiment corresponds one-to-one with each step in the automated testing method for intercom seats in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned automated testing method for intercom seats, and will not be repeated here.

[0247] Example 3

[0248] This embodiment provides an electronic device, including at least one processor and a memory. Optionally, the device further includes a communication component. The processor, memory, and communication component are connected via a bus.

[0249] In a specific implementation, at least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the above-described method.

[0250] The specific implementation process of the processor can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here.

[0251] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0252] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0253] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0254] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0255] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0256] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0257] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0258] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0259] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0260] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0261] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0262] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0263] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automated testing of intercom seats, characterized in that, include: Based on the logical attribute combination of each control in the control tree of the target software interface, the node-by-node comparison is performed with the pre-configured interference area matching rules. The screen area corresponding to the successfully matched control node is processed by pixel masking to obtain the denoised image to be recognized. A pre-made target control template image is obtained, and a matching algorithm based on similarity calculation is used on the denoised image to be identified to search for the region with the highest similarity to the target control template image, thereby obtaining the target detection region; At fixed time intervals, state matching detection, which includes three dimensions of color features, position offset and pattern contour, is performed simultaneously on the target detection area to obtain a sequence of multiple consecutive detection results. When the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, the data layer signal obtained by keyword matching from the background log of the target software system and the physical layer signal obtained by querying the underlying audio interface of the target software system are used to verify consistency and obtain the final test conclusion.

2. The method according to claim 1, characterized in that, The step involves comparing the logical attribute combinations of each control in the control tree of the target software interface with pre-configured interference area matching rules node by node, and performing pixel masking processing on the screen area corresponding to the successfully matched control node to obtain a denoised image to be identified, including: The complete control hierarchy and logical attributes of each control in the current interface of the intercom seat software are obtained through the system's underlying auxiliary function interface. An interface control tree model is constructed, and the logical attributes include at least one of control identifier, control name, control class name and parent control name. Load the pre-configured interference zone matching rule file. Each interference zone matching rule consists of at least one matching condition combination. Each matching condition includes an attribute name, a matching operator, and a target value. The matching operator can be any one of inclusion matching, exact matching, or regular expression matching. For each interference zone matching rule, based on the combination of multiple logical attribute matching conditions in the interference zone matching rule, the interface control tree model is recursively traversed and node-by-node logical attribute comparison is performed. Control nodes that simultaneously satisfy all combinations of conditions in the interference zone matching rule are determined as interference zone control nodes, and the real-time bounding box coordinates of the interference zone control nodes on the screen are read. Capture the original screen image of the current interface of the intercom seat software, and perform a masking operation on all pixels within the range defined by the real-time bounding box coordinates in the original screen image to obtain a denoised image to be identified.

3. The method according to claim 1, characterized in that, The process involves acquiring a pre-made target control template image, and then using a similarity-based matching algorithm on the denoised image to be recognized to search for the region with the highest similarity to the target control template image, thereby obtaining the target detection region. This includes: When the intercom software is in standby mode, a standard image of the target control to be detected is captured as the target control template image. The target control template image contains the complete outline and texture features of the target control. The normalized cross-correlation matching method is used to search for the region with the highest similarity to the target control template image on the denoised image to be identified, and the similarity score is calculated. When the similarity score is greater than or equal to a preset matching threshold, the region with the highest similarity score is determined as the target detection region; When the similarity score is less than the preset matching threshold, a positioning failure process is triggered, the positioning failure information is recorded, and the next frame image is obtained to re-position the image.

4. The method according to claim 1, characterized in that, The process involves simultaneously performing state matching detection on the target detection region at fixed time intervals, encompassing three dimensions: color features, positional offset, and pattern contour, to obtain a sequence of consecutive detection results, including: The state matching detection is repeatedly performed on the target detection area at a preset sampling time interval. The color feature detection method is as follows: extract the RGB values ​​of all pixels in the target control area of ​​the target control template image, calculate the mean and standard deviation of the R channel, G channel and B channel respectively, and set the color threshold range from the mean minus twice the standard deviation to the mean plus twice the standard deviation; extract the RGB values ​​of all pixels in the current target detection area, calculate the percentage of pixels falling within the color threshold range, and determine that the color dimension matching is successful when the percentage of pixels is greater than or equal to the preset color percentage threshold. The method for detecting the position offset is as follows: record the initial position coordinates of the target control template image when it is successfully matched for the first time; in each subsequent polling detection, calculate the pixel offset of the current matching position relative to the initial position coordinates; when the pixel offset is less than or equal to a preset pixel offset threshold, it is determined that the position dimension is successfully matched. The detection method for the pattern contour is as follows: extract the contour of the target control area in the target control template image, calculate multiple Hu moment feature values ​​of the contour, and form a standard shape feature vector; extract the contour of the target control area in the current target detection area and calculate the Hu moment feature values ​​to form a current shape feature vector; calculate the Euclidean distance between the standard shape feature vector and the current shape feature vector, and when the Euclidean distance is less than a preset similarity threshold, it is determined that the pattern dimension matching is successful; When the color dimension, the position dimension, and the pattern dimension are all determined to be a successful match, the single detection result is a successful match; otherwise, the single detection result is a failed match.

5. The method according to claim 4, characterized in that, The method further includes: Set a preset maximum timeout period, starting from the first execution of the state matching detection; When the cumulative execution time exceeds the preset maximum timeout time and the number of consecutive successful matches does not reach the stable number threshold, a timeout circuit breaker is triggered, polling is terminated, and a visual status signal of false is output. When the number of consecutive successful matches reaches the stable number threshold, the output visual state signal is true.

6. The method according to claim 1, characterized in that, When the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, the consistency of the data layer signal obtained by keyword matching from the target software system's background log and the physical layer signal obtained by querying the target software system's underlying audio interface is verified to obtain the final test conclusion, including: When the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, the visual state signal is determined to be true. The system monitors the background log stream output of the intercom system in real time. Regular expressions are used to match keywords in each line of the log stream. These regular expressions are predefined based on target events that correspond one-to-one with the interface state changes associated with the visual state signals. When a target event is matched within a preset monitoring time window, the output data layer signal is true; when the monitoring times out or no match is found in any logs, the output data layer signal is false. The target event is key log information in the background log that indicates a business operation has been successfully executed or its state has changed. The system queries the running status of the intercom seat audio driver process and the on / off status of the audio data stream by querying the underlying API of the operating system or the file system interface. When it is confirmed that the audio data stream is in a normal sending or receiving state, the physical layer signal is output as true; otherwise, the physical layer signal is output as false. The visual state signal, the data layer signal, and the physical layer signal are logically consistent. When all three are true, the final test result is "passed"; when all three are false, the final test result is "failed"; when the logical values ​​of the three are inconsistent, it is determined to be an abnormal conflict state.

7. The method according to claim 6, characterized in that, When the abnormal conflict state is determined, the following operations are performed automatically: Save a complete screenshot of the current in-call interface; Save a recent snapshot of the intercom seat system's logs; Record detailed data of the visual state signal, the data layer signal, and the physical layer signal; Generate conflict reports containing timestamps, signal details, and on-site data, and output anomaly conclusions for manual analysis.

8. An automated testing device for intercom seats, characterized in that, include: The noise reduction module is used to compare the logical attribute combination of each control in the control tree of the target software interface with the pre-configured interference area matching rules node by node, and perform pixel masking on the screen area corresponding to the successfully matched control node to obtain the noise-reduced image to be recognized. The matching module is used to acquire a pre-made target control template image, and to search for the region with the highest similarity to the target control template image on the denoised image to be identified using a similarity calculation-based matching algorithm to obtain the target detection region; The detection module is used to simultaneously perform state matching detection on the target detection area at fixed time intervals, including three dimensions: color features, position offset, and pattern contour, to obtain a sequence of multiple consecutive detection results. The verification module is used to verify the consistency of the data layer signal obtained by keyword matching from the background log of the target software system and the physical layer signal obtained by querying the underlying audio interface of the target software system when the number of consecutive successful matches in the detection result sequence reaches a preset stable number threshold, so as to obtain the final test conclusion.

9. An electronic device, characterized in that, include: At least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The method as described in any one of claims 1-7 is implemented when the computer program instructions are executed by the processor.

Images