Real-time rendering display system and method for UI interface layout information based on swan gap system

Abstract

The invention discloses a UI interface layout information real-time rendering display system and method based on a swan gap system, and relates to the field of man-machine interaction, and the method comprises the steps: obtaining an interface rendering data stream when a target application program runs in the swan gap system, extracting a layout metadata set from the interface rendering data stream, and displaying the layout metadata set in the swan gap system; the method comprises the following steps: determining a target element subset comprising percentage size attributes, a nested rolling container and a data binding relationship, and then generating a wireframe coverage layer with hierarchical transparency characteristics according to the target element subset; and displaying the hierarchical path information of the currently selected element, the pixel deviation ratio of the actual rendering size and the design draft size, the text content overflow state mark and the parent container constraint condition expression. According to the display system and method for real-time rendering of the user interface layout information based on the swan-monk system, the efficiency of repairing the user interface problem by a tester is improved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technologies, and particularly to a real-time rendering and display system and method for UI interface layout information based on the HarmonyOS system. Background Art

[0002] With the diversified development of intelligent terminal devices, application program interfaces need to be adapted to display devices with different screen sizes, resolutions, and aspect ratios. In the prior art, due to the coupling of layout constraint conditions and dynamic data binding, problems such as misalignment of interface elements, truncation of text content, and abnormal visual hierarchy coverage are likely to occur during multi-device rendering, resulting in a significant deviation between the display effect and the design draft.

[0003] Traditional adaptation and debugging methods rely on manual verification one by one on physical devices, and have defects such as low visualization degree of layout structures and poor traceability efficiency of abnormal problems. Especially for complex interfaces using percentage layouts and nested scroll containers, it is difficult for testers to perceive the mapping relationship between the rendering data stream and layout constraints in real time, resulting in lag in adaptation and optimization and inability to effectively prevent display conflict problems in multi-device environments. Summary of the Invention

[0004] To overcome the above deficiencies of the prior art, this application provides a real-time rendering and display system and method for UI interface layout information based on the HarmonyOS system.

[0005] In a first aspect, a real-time rendering and display method for UI interface layout information based on the HarmonyOS system provided by this application includes: Obtain the interface rendering data stream of a target application when running on the HarmonyOS system, and extract a layout metadata set including element type identifiers, hierarchical depth values, absolute coordinate ranges, dynamic constraint expressions, and associated resource identifiers from the interface rendering data stream through the underlying interface of the HarmonyOS system; Perform topological sorting and sensitive element marking processing on the layout metadata set to determine a target element subset including percentage size attributes, nested scroll containers, and data binding relationships; Map the target element subset to the spatial coordinate system of the display interface to generate a wireframe overlay layer with hierarchical transparency characteristics, where the color of the wireframes in the wireframe overlay layer fades according to a preset color scale rule based on the hierarchical depth of the target elements in the target element subset, and the width of the wireframes is inversely proportional to the area of the display regions of the target elements; Generate an interactive layout analysis panel on the side of the display interface. When it is detected that a tester clicks on the wireframe overlay layer, display the hierarchical path information of the currently selected element, the pixel deviation rate between the actual rendering size and the design draft size, the text content overflow status mark, and the parent container constraint condition expression; Based on the screen pixel density, resolution parameters, and safety display area boundary information of the target device, calculate the deviation value between the actual rendering ratio and the preset standard ratio of each target element in the subset of target elements. When the deviation value exceeds the first preset threshold, generate an adaptation exception warning message and mark the potential area with display conflicts.

[0006] Preferably, intercept the layer composition data of the frame buffer through the graphics composition service interface of the HarmonyOS system; Analyze the geometric vertex coordinates and texture mapping parameters of each drawing instruction in the layer composition data; For elements with dynamic constraint conditions, extract the anchor connection relationship and relative weight distribution ratio between them and the parent container; Convert the analysis results into a structured data set including a globally unique identifier, control type code, absolute coordinate range matrix, hierarchical depth index value, dynamic constraint condition logic expression, and associated resource file feature hash value.

[0007] Preferably, set the transparency decreasing gradient according to the hierarchical depth value, so that each increase in the hierarchical depth corresponds to a reduction in the layer transparency by a preset percentage; Add a periodic visual warning mark to the target element with text overflow. The display frequency of the warning mark is positively correlated with the text overflow amount; Draw a warning boundary with a dotted line style around the boundary of the target element with dynamic constraint conditions. The interval distance of the dotted line is dynamically adjusted according to the change rate of the constraint conditions; Use the multi-sampling anti-aliasing algorithm to smooth the wireframe edges and overlay a text label containing the type abbreviation program and unique identifier at the preset position of each target element.

[0008] Preferably, when it is detected that the line height value of the text element is less than the preset multiple of the font size, mark it as an abnormal line spacing compression; For elements using the absolute positioning method, calculate the offset distance between its center point coordinates and the ideal layout grid node. When the offset distance exceeds the second preset threshold, mark it as an abnormal positioning deviation; When the cumulative size of the child elements in the scrolling container exceeds the preset multiple of the visible area size, trigger a scrolling performance degradation warning; Detect the scaling mode of the image element. When it is detected that there is forced stretching and the aspect ratio change exceeds the third preset threshold, mark it as an abnormal image distortion.

[0009] Preferably, establish a wireless debugging channel with multiple HarmonyOS test devices to obtain the interface rendering data and layout analysis data of multiple HarmonyOS test devices in real time; A comparison view of multiple HarmonyOS test devices is displayed side by side on the display interface, and the difference area markers of the wireframe layers of multiple HarmonyOS test devices are superimposed and displayed on the comparison view; When it is detected that a specific device has a layout anomaly, a set of screen parameters of the specific device is extracted, and historical optimization solutions are matched based on parameter similarity to generate layout adaptation suggestions.

[0010] Preferably, a visual acquisition device captures the visual focus coordinates of the tester at a preset sampling interval; According to the mapping relationship between the visual focus coordinates and the wireframe, a first transparency adjustment strategy is implemented for the wireframe within a preset radius centered on the coordinates, and a second transparency adjustment strategy is implemented for the wireframes in other areas; When it is detected that the tester's fixation duration on the same element exceeds the first preset time threshold, the historical change record of the element's constraint conditions is displayed on the interactive layout analysis panel, and the position of the corresponding layout source program in the integrated development environment is located; Based on the visual focus distribution data, an interface layout complexity index is calculated. When the index exceeds the preset complexity threshold, an optimization suggestion to split the composite layout into atomic components is generated.

[0011] Preferably, a version control database is established to record the wireframe snapshot data, device adaptation reports, and corresponding source program change records after each layout modification; When it is detected that the similarity between the current layout defect features and historical cases exceeds the preset similarity threshold, the associated historical repair solutions and verification test case sets are pushed; A layout defect prediction model based on a graph neural network is constructed. The current view tree structure data and the device parameter set are input into the prediction model, and potential defect type identifiers and their confidence scores are output; For the defect types with confidence scores exceeding the preset confidence threshold, a repair patch program containing constraint condition optimization rules and container logic improvement solutions is generated; The compatibility and performance improvement indicators of the repair patch program are verified in an isolated sandbox environment, and after passing the verification, it is deployed to the target device through the hot update mechanism.

[0012] In a second aspect, a real-time rendering and display system for UI interface layout information based on the HarmonyOS system includes: A metadata extraction unit, which is used to obtain the interface rendering data stream when the target application program runs on the HarmonyOS system, and extract a set of layout metadata including element type identifiers, hierarchical depth values, absolute coordinate ranges, dynamic constraint expressions, and associated resource identifiers from the interface rendering data stream through the underlying interface of the HarmonyOS system; A target element subset determination unit for performing topological sorting and sensitive element marking processing on a layout metadata set to determine a target element subset that includes percentage size attributes, nested scroll containers, and data binding relationships; A coordinate mapping unit for performing a spatial coordinate system mapping between the target element subset and the display interface to generate a wireframe overlay layer with hierarchical transparency characteristics, where the color of the wireframe in the wireframe overlay layer fades according to a preset color scale rule based on the hierarchical depth of the target elements in the target element subset, and the width of the wireframe is inversely proportional to the area of the display area of the target element; An analysis panel generation unit for generating an interactive layout analysis panel on the side of the display interface, and when detecting a click operation by a tester on the wireframe overlay layer, displaying the hierarchical path information of the currently selected element, the pixel deviation rate between the actual rendered size and the design draft size, the text content overflow status mark, and the parent container constraint condition expression; A warning unit for calculating the deviation value between the actual rendering ratio of each target element in the target element subset and the preset standard ratio based on the screen pixel density, resolution parameters, and safe display area boundary information of the target device, and generating an adaptation exception warning message and marking potential areas with display conflicts when the deviation value exceeds the first preset threshold.

[0013] Compared with the prior art, the present invention has the following features and beneficial effects: By extracting the layout metadata set in the interface rendering data stream in real time, the limitation of traditional debugging tools relying on manual program review is solved, ensuring the acquisition of complete layout constraint relationships and hierarchical information, and providing a reliable data basis for anomaly detection.

[0014] Based on topological sorting and sensitive element marking to screen the target element subset, effectively focusing on key elements such as percentage sizes and nested containers with adaptation risks, avoiding resource waste in full-scale analysis, and significantly improving the problem location efficiency.

[0015] The hierarchical wireframe overlay layer generated through spatial coordinate system mapping intuitively presents the element distribution density and nesting depth with color fading and dynamic line width, reducing the visual cognitive load of testers and achieving rapid understanding of the layout structure.

[0016] Combined with the real-time feedback mechanism of the interactive analysis panel, when a wireframe click operation is detected, the element deviation rate, constraint conditions, and design draft comparison data are dynamically associated and displayed, forming a closed-loop optimization process of "visual positioning - data diagnosis - program adjustment", which significantly shortens the adaptation and debugging cycle. In addition, based on the dynamic ratio deviation calculation model of device parameters, it can early warn of adaptation anomalies during the layout rendering stage, guide targeted optimization by marking potential conflict areas, prevent display problems such as interface deformation and content occlusion from the root cause, and ensure the consistency of the interface rendering effect in a multi-device environment. Brief Description of the Drawings

[0017] Figure 1 It is a step block diagram of a method for real-time rendering and display of UI interface layout information based on the HarmonyOS system mainly embodied in this embodiment.

[0018] Figure 2 It is a structural block diagram of a system for real-time rendering and display of UI interface layout information based on the HarmonyOS system mainly embodied in this embodiment. Detailed Description of the Preferred Embodiments

[0019] The present invention will be further described in detail below in conjunction with the following embodiments.

[0020] Refer to Figure 1 , a method for real-time rendering and display of UI interface layout information based on the HarmonyOS system, the method includes the following steps: S1. Obtain the interface rendering data stream of the target application when running on the HarmonyOS system, and extract a layout metadata set including element type identifiers, hierarchical depth values, absolute coordinate ranges, dynamic constraint expressions, and associated resource identifiers from the interface rendering data stream through the underlying interface of the HarmonyOS system.

[0021] S2. Perform topological sorting and sensitive element marking processing on the layout metadata set to determine a target element subset including percentage size attributes, nested scroll containers, and data binding relationships.

[0022] S3. Map the target element subset to the spatial coordinate system of the display interface to generate a wireframe overlay layer with hierarchical transparency characteristics, where the color of the wireframes in the wireframe overlay layer fades according to a preset color scale rule based on the hierarchical depth of the target elements in the target element subset, and the width of the wireframes is inversely proportional to the area of the display area of the target elements.

[0023] S4. Generate an interactive layout analysis panel on the side of the display interface. When a click operation by a tester on the wireframe overlay layer is detected, display the hierarchical path information of the currently selected element, the pixel deviation rate between the actual rendered size and the design draft size, the text content overflow status mark, and the parent container constraint condition expression.

[0024] S5. Calculate the deviation value between the actual rendering ratio and the preset standard ratio of each target element in the target element subset based on the screen pixel density, resolution parameters, and secure display area boundary information of the target device. When the deviation value exceeds the first preset threshold, generate an adaptation exception warning message and mark the potential area with display conflicts.

[0025] Specifically, first, by extracting the layout metadata set in the interface rendering data stream in real time, the limitations of traditional debugging tools relying on manual program review are solved, ensuring the acquisition of complete layout constraint relationships and hierarchical information, and providing a reliable data basis for anomaly detection. Second, based on topological sorting and sensitive element marking, the target element subset is filtered, effectively focusing on key elements such as percentage sizes and nested containers with adaptation risks, avoiding resource waste in full-scale analysis, and significantly improving the problem location efficiency. Further, through the hierarchical wireframe overlay layer generated by spatial coordinate system mapping, the element distribution density and nesting depth are visually presented with color gradients and dynamic line widths, reducing the visual cognitive load of testers and enabling rapid understanding of the layout structure. Combined with the real-time feedback mechanism of the interactive analysis panel, when a wireframe click operation is detected, the element deviation rate, constraint conditions, and design draft comparison data are dynamically associated and displayed, forming a closed-loop optimization process of "visual positioning - data diagnosis - program adjustment", significantly shortening the adaptation debugging cycle. In addition, based on the dynamic ratio deviation calculation model of device parameters, it is possible to early warn of adaptation anomalies during the layout rendering stage, guide targeted optimization by marking potential conflict areas, and prevent display problems such as interface deformation and content occlusion from the source, ensuring the consistency of interface rendering effects in a multi-device environment.

[0026] Specific step S1 includes the following sub-steps: Intercept the layer composition data of the frame buffer through the graphics composition service interface of the HarmonyOS system; Parse the geometric vertex coordinates and texture mapping parameters of each drawing instruction in the layer composition data; For elements containing dynamic constraint conditions, extract the anchor connection relationship and relative weight allocation ratio between them and the parent container; Convert the parsing results into a structured data set containing globally unique identifiers, control type codes, absolute coordinate range matrices, hierarchical depth index values, dynamic constraint condition logical expressions, and associated resource file feature hash values.

[0027] Specifically, first, through the graphics composition service interface of the HarmonyOS, it is possible to capture the layer composition data of the frame buffer in real time to ensure obtaining real-time information on interface rendering. Next, each drawing instruction in these layer composition data is parsed to extract geometric vertex coordinates and texture mapping parameters, which can help understand how each element is drawn and its position information. For elements with dynamic constraint conditions, the system can further extract the anchor connection relationship and relative weight allocation ratio between them and the parent container, which can help understand the dynamic adjustment behavior of the elements in the interface. Finally, all these parsed results are converted into a structured data set, including a globally unique identifier, a control type code, an absolute coordinate range matrix, a hierarchical depth index value, a dynamic constraint condition logic expression, and an associated resource file feature hash value. This data set provides a solid foundation for subsequent layout analysis and anomaly detection. For example, in an actual application scenario, the system captures the layer composition data of the frame buffer, parses the geometric vertex coordinates and texture mapping parameters of a certain button element, and discovers that there is an anomaly in the anchor connection relationship between the dynamic constraint conditions of the button element and the parent container. The system can convert this information into a structured data set for analysis and processing in subsequent steps.

[0028] Specifically, the process of step S2 can be to perform topological sorting and sensitive element marking processing on the layout metadata set to determine a subset of target elements including percentage size attributes, nested scroll containers, and data binding relationships. Specifically, first, the system can perform topological sorting on the layout metadata set to ensure that the hierarchical and dependency relationships between elements are correctly processed. Next, the system can perform marking processing on sensitive elements, which usually include elements with percentage size attributes, nested scroll containers, and data binding relationships. These elements have a relatively high adaptation risk in the interface layout. By filtering out this subset of target elements, the system can focus on key elements, avoid wasting resources in full-scale analysis, and significantly improve the problem location efficiency. For example, in an actual application scenario, through topological sorting and sensitive element marking processing, the system discovers that there is an anomaly in the percentage size attribute of a certain nested scroll container. The system can mark this container as part of the subset of target elements for further analysis and processing in subsequent steps.

[0029] Specific steps of S3 include the following sub-steps: Set a transparency decreasing gradient according to the hierarchical depth value, so that for each increase in the hierarchical depth level, the layer transparency is reduced by a preset percentage; Add a periodic visual warning mark to the target element with text overflow, and the display frequency of the warning mark is positively correlated with the amount of text overflow; Draw a warning boundary in a dotted line style around the boundary of the target element with dynamic constraint conditions, and the interval distance of the dotted lines is dynamically adjusted according to the change rate of the constraint conditions; Use the multi-sampling anti-aliasing algorithm to smooth the wireframe edges, and superimpose and display text labels containing type abbreviation programs and unique identifiers at preset positions of each target element.

[0030] Specifically, the system can set a decreasing gradient of transparency according to the hierarchical depth value, so that each increase in the hierarchical depth corresponds to a decrease in the transparency of the layer by a preset percentage, which can visually present the nesting depth and distribution density of the elements. Then, for target elements with text overflow, the system can add periodic visual warning marks, and the display frequency of the warning marks is positively correlated with the amount of text overflow, which can help testers quickly identify text overflow problems. For target elements with dynamic constraint conditions, the system can draw a warning boundary in a dotted line style around their boundaries, and the interval distance of the dotted lines is dynamically adjusted according to the change rate of the constraint conditions, which can help testers understand the dynamic adjustment behavior of the elements. Finally, the system can use the multi-sampling anti-aliasing algorithm to smooth the wireframe edges, and superimpose and display text labels containing type abbreviation programs and unique identifiers at preset positions of each target element, which can improve the readability and recognition of the wireframe. For example, in an actual application scenario, by setting the decreasing gradient of transparency, the system finds that an element with a relatively deep nesting level has a low transparency. The system can add periodic visual warning marks and draw a warning boundary in a dotted line style around its boundary for testers to further analyze and process.

[0031] Specifically, the process of step S4 can be as follows: An interactive layout analysis panel is generated on the side of the display interface. When it detects a click operation by the tester on the wireframe overlay layer, it displays the hierarchical path information of the currently selected element, the pixel deviation rate between the actual rendered size and the design draft size, the text content overflow status flag, and the parent container constraint condition expression. Specifically, first, the system can generate an interactive layout analysis panel on the side of the display interface, and this panel can display the detailed information of the currently selected element in real time. When it detects a click operation by the tester on the wireframe overlay layer, the system can display the hierarchical path information of the currently selected element to help the tester understand the nesting relationship of the elements. Then, the system can display the pixel deviation rate between the actual rendered size and the design draft size to help the tester identify size deviation problems. For a target element with text overflow, the system can display the text content overflow status flag to help the tester quickly identify text overflow problems. Finally, the system can display the parent container constraint condition expression to help the tester understand the dynamic adjustment behavior of the elements. For example, in an actual application scenario, by generating an interactive layout analysis panel, the system finds that there is a large deviation between the actual rendered size and the design draft size of a certain button element, and the system can display the hierarchical path information and the parent container constraint condition expression of this button element for the tester to further analyze and process.

[0032] Specific steps S5 include the following sub-steps: When it detects that the line height value of a text element is less than a preset multiple of the font size, it is marked as an abnormal line spacing compression; For an element using the absolute positioning method, calculate the offset distance between its center point coordinates and the ideal layout grid node. When the offset distance exceeds the second preset threshold, it is marked as an abnormal positioning deviation; When the cumulative size of the child elements in a scroll container exceeds a preset multiple of the visible area size, a scroll performance degradation warning is triggered; Detect the scaling mode of an image element. When it detects forced stretching and the aspect ratio change exceeds the third preset threshold, it is marked as an abnormal image distortion.

[0033] Specifically, the system can detect whether the line height value of a text element is less than a preset multiple of the font size. If it is less, it is marked as an abnormal line spacing compression, which can help testers identify text line spacing problems. Next, for elements using the absolute positioning method, the system can calculate the offset distance between the center point coordinates of the element and the ideal layout grid node. If the offset distance exceeds a second preset threshold, it is marked as an abnormal positioning deviation, which can help testers identify positioning deviation problems. For the case where the cumulative size of the child elements within a scroll container exceeds a preset multiple of the visible area size, the system can trigger a warning for deteriorated scrolling performance, which can help testers identify scrolling performance problems. Finally, the system can detect the scaling mode of an image element. If it is detected that the image is forced to stretch and the aspect ratio changes by more than a third preset threshold, it is marked as an abnormal image distortion, which can help testers identify image distortion problems. For example, in an actual application scenario, by detecting the line height value of a text element, the system finds that the line height value of a certain text element is less than the preset multiple of the font size, and the system can mark this text element as an abnormal line spacing compression for further analysis and processing by testers.

[0034] In some embodiments, the real-time rendering and display method for the UI interface layout information based on the HarmonyOS may further include the following steps: Establish a wireless debugging channel with multiple HarmonyOS test devices to obtain the interface rendering data and layout parsing data of the multiple HarmonyOS test devices in real time; Display the comparison views of the multiple HarmonyOS test devices side by side on the display interface, and the comparison views superimpose and display the marked difference areas of the wireframe layers of the multiple HarmonyOS test devices; When it is detected that a specific device has a layout anomaly, extract the set of screen parameters of the specific device, match the historical optimization solutions based on the parameter similarity, and generate layout adaptation suggestions.

[0035] Specifically, the system can establish a wireless debugging channel with multiple HarmonyOS test devices to obtain the interface rendering data and layout parsing data of these devices in real time, ensuring that real-time information of multiple devices is obtained. Next, the system can display the comparison views of the multiple HarmonyOS test devices side by side on the display interface, and the comparison views will superimpose and display the marked difference areas of the wireframe layers of the multiple HarmonyOS test devices, which can help testers quickly identify the layout differences between multiple devices. When it is detected that a specific device has a layout anomaly, the system can extract the set of screen parameters of the device, match the historical optimization solutions based on the parameter similarity, and generate layout adaptation suggestions, which can help testers quickly solve the layout anomaly problem. For example, in an actual application scenario, by establishing a wireless debugging channel, the system discovers that there is an anomaly in the interface rendering data of a certain HarmonyOS test device, and the system can extract the set of screen parameters of the device and generate layout adaptation suggestions for further analysis and processing by testers.

[0036] In some embodiments, the method for real-time rendering and display of the UI interface layout information based on the HarmonyOS may further include the following steps: Capture the visual focus coordinates of the tester at a preset sampling interval through a visual acquisition device; According to the mapping relationship between the visual focus coordinates and the wireframe, implement a first transparency adjustment strategy for the wireframes within a preset radius range centered on the coordinates, and implement a second transparency adjustment strategy for the wireframes in other areas; When it is detected that the tester's fixation duration on the same element exceeds the first preset time threshold, display the historical change record of the element's constraint conditions on the interactive layout analysis panel, and locate the corresponding layout source program position in the integrated development environment; Calculate the interface layout complexity index based on the visual focus distribution data, and when the index exceeds the preset complexity threshold, generate an optimization suggestion to split the composite layout into atomic components.

[0037] Specifically, the system can capture the visual focus coordinates of the tester at a preset sampling interval through a visual acquisition device to ensure obtaining the visual focus information of the tester. Then, according to the mapping relationship between the visual focus coordinates and the wireframe, the system can implement a first transparency adjustment strategy for the wireframes within a preset radius range centered on the coordinates, and implement a second transparency adjustment strategy for the wireframes in other areas, which can help the tester quickly focus on the key areas. When it is detected that the tester's fixation duration on the same element exceeds the first preset time threshold, the system can display the historical change record of the element's constraint conditions on the interactive layout analysis panel and locate the corresponding layout source program position in the integrated development environment, which can help the tester quickly understand the adjustment history of the element. Finally, the system can calculate the interface layout complexity index based on the visual focus distribution data. If the index exceeds the preset complexity threshold, an optimization suggestion to split the composite layout into atomic components will be generated, which can help the tester optimize the interface layout. For example, in an actual application scenario, the system captures the visual focus coordinates of the tester and finds that the tester's fixation duration on a certain element exceeds the preset threshold. The system can display the historical change record of the element's constraint conditions and generate an optimization suggestion for the tester to further analyze and process.

[0038] In some embodiments, the method for real-time rendering and display of the UI interface layout information based on the HarmonyOS may further include the following steps: Establish a version control database to record the wireframe snapshot data, device adaptation report, and the corresponding source program change record after each layout modification; When it is detected that the similarity between the current layout defect feature and the historical cases exceeds the preset similarity threshold, push the associated historical repair solutions and verification test case sets; Build a layout defect prediction model based on a graph neural network, input the current view tree structure data and the set of device parameters into the prediction model, and output the potential defect type identifier and its confidence score; For defect types with a confidence score exceeding the preset confidence threshold, generate a repair patch program containing constraint condition optimization rules and container logic improvement solutions; Verify the compatibility and performance improvement metrics of the repair patch program in an isolated sandbox environment, and deploy it to the target device through a hot update mechanism after verification.

[0039] Specifically, the system can establish a version control database to record the wireframe snapshot data, device adaptation reports, and corresponding source program change records after each layout modification, ensuring that complete historical information about layout modifications is obtained. Then, when the similarity between the current layout defect features and historical cases exceeds the preset similarity threshold, the system can push associated historical repair solutions and verification test case sets, which can help testers quickly solve similar problems. The system also builds a layout defect prediction model based on a graph neural network, inputs the current view tree structure data and the set of device parameters into the prediction model, and outputs the potential defect type identifier and its confidence score, which can help testers identify potential problems in advance. For defect types with a confidence score exceeding the preset confidence threshold, the system can generate a repair patch program containing constraint condition optimization rules and container logic improvement solutions, which can help testers quickly fix problems. Finally, the system can verify the compatibility and performance improvement metrics of the repair patch program in an isolated sandbox environment and deploy it to the target device through a hot update mechanism after verification, which can ensure the security and effectiveness of the repair patch program. For example, in an actual application scenario, the system establishes a version control database, discovers that the similarity between the current layout defect features and historical cases exceeds the preset threshold, and the system can push associated historical repair solutions and generate a repair patch program for testers to further analyze and process.

[0040] A real-time rendering and display system for UI interface layout information based on the HarmonyOS system, by applying a method for real-time rendering and display of UI interface layout information based on the HarmonyOS system as described above, including a metadata extraction unit, a target element subset determination unit, a coordinate mapping unit, an analysis panel generation unit, and a warning unit, referring to Figure 2, obtain the interface rendering data stream of the target application when it runs on the HarmonyOS system through the metadata extraction unit, and extract a layout metadata set including element type identifiers, hierarchical depth values, absolute coordinate ranges, dynamic constraint expressions, and associated resource identifiers from the interface rendering data stream through the underlying interface of the HarmonyOS system; perform topological sorting and sensitive element marking processing on the layout metadata set through the target element subset determination unit to determine a target element subset including percentage size attributes, nested scroll containers, and data binding relationships; map the target element subset to the display interface through the coordinate mapping unit to generate a wireframe overlay layer with hierarchical transparency characteristics, where the color of the wireframe in the wireframe overlay layer fades according to a preset color scale rule based on the hierarchical depth of the target element in the target element subset, and the width of the wireframe is inversely proportional to the area of the display area of the target element; generate an interactive layout analysis panel on the side of the display interface through the analysis panel generation unit, and when it detects a click operation by the tester on the wireframe overlay layer, display the hierarchical path information of the currently selected element, the pixel deviation rate between the actual rendering size and the design draft size, the text content overflow status mark, and the parent container constraint condition expression; based on the screen pixel density, resolution parameters, and safe display area boundary information of the target device, calculate the deviation value between the actual rendering ratio and the preset standard ratio of each target element in the target element subset through the warning unit, and generate an adaptation exception warning message and mark the potential area with a display conflict when the deviation value exceeds the first preset threshold.

[0041] The above are all preferred embodiments of this application. Without limiting the protection scope of this application accordingly, therefore: All equivalent changes made according to the structure, shape, and principle of this application shall be covered within the protection scope of this application.

Claims

1. A method for real-time rendering and displaying UI interface layout information based on Hongmeng system, characterized in that: The following steps are involved: Obtain an interface rendering data stream of a target application when it is running on the Hongmeng system, and extract a layout metadata set including an element type identifier, a layer depth value, an absolute coordinate range, a dynamic constraint expression, and an associated resource identifier from the interface rendering data stream through an underlying interface of the Hongmeng system; Performing topological sorting and sensitive element marking processing on the layout metadata set to determine a target element subset including percentage size attributes, nested scroll containers, and data binding relationships; Mapping the target element subset to the display interface in a spatial coordinate system to generate a wireframe overlay layer with hierarchical transparency characteristics, wherein the color of the wireframe in the wireframe overlay layer changes gradually according to a preset color scale rule based on the hierarchical depth of the target element in the target element subset, and the width of the wireframe is inversely proportional to the area of ​​the display region of the target element; An interactive layout analysis panel is generated on the side of the display interface. When a click operation of the tester on the wireframe overlay layer is detected, the hierarchical path information of the currently selected element, the pixel deviation rate between the actual rendering size and the design draft size, the text content overflow status mark, and the parent container constraint condition expression are displayed; Based on the screen pixel density, resolution parameters and safe display area boundary information of the target device, the deviation value between the actual rendering ratio of each target element in the target element subset and the preset standard ratio is calculated. When the deviation value exceeds the first preset threshold, an adaptation abnormality alarm message is generated and the potential area where the display conflict exists is marked.

2. According to the method of real-time rendering and displaying UI interface layout information based on Hongmeng system in claim 1, it is characterized in that: The steps of obtaining the interface rendering data stream of the target application when the Hongmeng system is running, and extracting a layout metadata set including an element type identifier, a hierarchical depth value, an absolute coordinate range, a dynamic constraint expression, and an associated resource identifier from the interface rendering data stream through the underlying interface of the Hongmeng system are specifically as follows: Intercepting the layer synthesis data of the frame buffer through the graphics synthesis service interface of the Hongmeng system; Parsing the geometric vertex coordinates and texture mapping parameters of each drawing instruction in the layer synthesis data; For elements containing dynamic constraints, extract the anchor point connection relationship between them and the parent container and the relative weight distribution ratio; Convert the parsing result into a structured data set containing a globally unique identifier, a control type code, an absolute coordinate range matrix, a hierarchical depth index value, a dynamic constraint condition logic expression, and a characteristic hash value of the associated resource file.

3. According to claim 2, a method for real-time rendering and displaying UI interface layout information based on Hongmeng system is characterized in that: The steps of mapping the target element subset to the real-time screen image in a spatial coordinate system to generate a wireframe overlay layer with hierarchical transparency characteristics are specifically as follows: According to the layer depth value, a transparency decreasing gradient is set, so that each increase in layer depth corresponds to a decrease in layer transparency by a preset percentage; Adding a periodic visual warning mark to the target element with text overflow, wherein the display frequency of the warning mark is positively correlated with the amount of text overflow; Draw a warning boundary in the form of a dotted line around the boundary of the target element with dynamic constraints, where the interval between the dotted lines is dynamically adjusted according to the rate of change of the constraints; The wireframe edges are smoothed using a multi-sampling anti-aliasing algorithm, and a text label containing the type abbreviation and a unique identifier is superimposed and displayed at a preset position on each target element.

4. According to claim 3, a method for real-time rendering and displaying UI interface layout information based on Hongmeng system is characterized in that: Based on the screen pixel density, resolution parameters and safe display area boundary information of the target device, the deviation value between the actual rendering ratio of each interface element and the preset standard ratio is calculated, and when the deviation value exceeds a first preset threshold, an adaptation abnormality warning message is generated and a potential area with display conflict is marked, specifically, the steps are as follows: When it is detected that the line height value of a text element is less than a preset multiple of the font size, it is marked as line spacing compression abnormality; For elements using absolute positioning, calculate the offset distance between the coordinates of the center point and the ideal layout grid node, and mark it as positioning deviation abnormality when the offset distance exceeds a second preset threshold; When the cumulative size of the sub-elements in the scroll container exceeds the preset multiple of the visible area size, a scroll performance degradation warning is triggered; The scaling mode of the image element is detected, and when forced stretching is detected and the aspect ratio change exceeds a third preset threshold, it is marked as image distortion abnormality.

5. According to the method of real-time rendering and displaying UI interface layout information based on Hongmeng system in claim 1, it is characterized in that: The method further comprises: Establish a wireless debugging channel with multiple Hongmeng test devices to obtain interface rendering data and layout analysis data of the multiple Hongmeng test devices in real time; Displaying comparison views of the multiple Hongmeng test devices side by side on the display interface, the comparison views superimposedly displaying difference area marks of the wireframe layers of the multiple Hongmeng test devices; When a layout anomaly is detected on a specific device, a screen parameter set of the specific device is extracted, historical optimization solutions are matched based on parameter similarity, and layout adaptation suggestions are generated.

6. According to a method for real-time rendering and displaying UI interface layout information based on Hongmeng system according to claim 1, it is characterized in that: The method further comprises: The visual focus coordinates of the test person are captured at a preset sampling interval by a visual acquisition device; According to the mapping relationship between the visual focus coordinates and the wireframe, a first transparency adjustment strategy is implemented for the wireframe within a preset radius centered on the coordinates, and a second transparency adjustment strategy is implemented for the wireframe in other areas; When it is detected that the tester's gaze time on the same element exceeds a first preset time threshold, the constraint condition history change record of the element is displayed on the interactive layout analysis panel, and the corresponding layout source program position in the integrated development environment is located; An interface layout complexity index is calculated based on the visual focus distribution data, and when the index exceeds a preset complexity threshold, an optimization suggestion is generated to split the complex layout into atomic components.

7. According to the method of real-time rendering and displaying of UI interface layout information based on Hongmeng system in claim 1, it is characterized in that: The method further comprises: Establish a version control database to record wireframe snapshot data, device adaptation reports and corresponding source program change records after each layout modification; When it is detected that the similarity between the current layout defect feature and the historical case exceeds the preset similarity threshold, the associated historical repair solution and verification test case set are pushed; Construct a layout defect prediction model based on a graph neural network, input the current view tree structure data and the device parameter set into the prediction model, and output the potential defect type identification and its confidence score; For defect types whose confidence scores exceed the preset confidence threshold, a repair patch program containing constraint optimization rules and container logic improvement solutions is generated; The compatibility and performance improvement indicators of the repair patch are verified in an isolated sandbox environment, and after passing the verification, it is deployed to the target device through a hot update mechanism.

8. A real-time rendering and display system for UI interface layout information based on Hongmeng system, characterized in that: The system is used to implement the real-time rendering and display method of UI interface layout information based on the Hongmeng system according to any one of claims 1 to 7, including: A metadata extraction unit, used to obtain an interface rendering data stream of a target application when the target application is running on the Hongmeng system, and extract a layout metadata set including an element type identifier, a layer depth value, an absolute coordinate range, a dynamic constraint expression, and an associated resource identifier from the interface rendering data stream through an underlying interface of the Hongmeng system; A target element subset determination unit, used to perform topological sorting and sensitive element marking processing on the layout metadata set, and determine a target element subset including percentage size attributes, nested scroll containers, and data binding relationships; A coordinate mapping unit, used to map the target element subset with the display interface in a spatial coordinate system, and generate a wireframe overlay layer with a hierarchical transparency feature, wherein the color of the wireframe in the wireframe overlay layer changes gradually according to a preset color scale rule based on the hierarchical depth of the target element in the target element subset, and the width of the wireframe is inversely proportional to the area of ​​the display region of the target element; An analysis panel generation unit is used to generate an interactive layout analysis panel on the side of the display interface, and when a click operation of the tester on the wireframe overlay layer is detected, the hierarchical path information of the currently selected element, the pixel deviation rate between the actual rendering size and the design draft size, the text content overflow status mark, and the parent container constraint condition expression are displayed; A warning unit is used to calculate the deviation value between the actual rendering ratio of each target element in the target element subset and the preset standard ratio based on the screen pixel density, resolution parameters and safe display area boundary information of the target device, and generate adaptation abnormality warning information and mark the potential area where display conflict exists when the deviation value exceeds a first preset threshold.

Images