User interface adjustment method and apparatus, storage medium, and electronic device

Abstract

The application discloses a user interface adjustment method and device, a storage medium and an electronic device, and relates to the technical field of smart homes. The user interface adjustment method comprises the following steps: constructing a second user interface conforming to an expected standard according to image data features in an actual first user interface; calculating mean square errors of a plurality of pixels according to a plurality of first pixels in the first user interface and a plurality of second pixels in the second user interface, wherein the plurality of pixels comprise the plurality of first pixels and the plurality of second pixels; and in the case that the mean square errors are greater than a preset threshold, determining that the first user interface is abnormal, and adjusting the first user interface according to an abnormal reason of the abnormality. By using the technical scheme, the problem that, in the related art, a traditional UI abnormality detection scheme relies on simple rules and limited templates to identify rendering abnormalities of a UI, thereby resulting in a low accuracy of abnormality detection, is solved.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a method and apparatus for adjusting a user interface, a storage medium, and an electronic device. Background Technology

[0002] In current smart home scenarios involving cross-brand device interaction, users often control smart devices from different manufacturers via mobile terminals or Augmented Reality Interfaces (AR). The user interface (UI) is typically dynamically rendered by the Android system or WebView components. Due to the fragmentation of the Android ecosystem, significant differences exist between devices from different brands in UI layout, control styles, rendering engines, and system versions, leading to frequent rendering anomalies in interface elements (such as button offsets, missing icons, overlapping text, and control occlusion). Existing technologies mostly employ static detection methods based on rule matching or template comparison. This involves pre-constructing a limited number of standard UI templates and judging anomalies through image similarity calculations or control attribute comparisons, which is highly prone to missed and false detections.

[0003] In response to the problem that traditional UI anomaly detection schemes rely on simple rules and limited templates to identify UI rendering anomalies, resulting in low accuracy of anomaly detection, no effective solution has yet been proposed. Summary of the Invention

[0004] This application provides a method and apparatus for adjusting a user interface, a storage medium, and an electronic device, to at least solve the problem in the related art that traditional UI anomaly detection schemes rely on simple rules and limited templates to identify UI rendering anomalies, resulting in low accuracy of anomaly detection.

[0005] According to one embodiment of this application, a method for adjusting a user interface is provided, comprising: constructing a second user interface conforming to an expected standard based on image data features in an actual first user interface; calculating the mean square error of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include the multiple first pixels and the multiple second pixels; determining that the first user interface is abnormal when the mean square error is greater than a preset threshold, and adjusting the first user interface according to the cause of the abnormality.

[0006] In an optional embodiment, after adjusting the first user interface according to the cause of the anomaly, the method further includes: acquiring light intensity data of the area where the first terminal device is located, wherein the first user interface is used to display the status of the first terminal device; determining that the light intensity data belongs to a weak light range when the light intensity data is less than a first preset value; determining that the light intensity data belongs to a medium light range when the light intensity data is greater than or equal to the first preset value and less than a second preset value; determining that the light intensity data belongs to a strong light range when the light intensity data is greater than or equal to the second preset value; adjusting the recognition threshold of a plurality of first controls in a third user interface according to the light intensity range to which the light intensity data belongs, wherein the recognition threshold is the minimum reference value for recognizing the plurality of first controls, the light intensity range includes: the weak light range, the medium light range, and the strong light range, and the third user interface is the adjusted first user interface; and adjusting the visibility of the plurality of first controls in the third user interface according to the recognition threshold.

[0007] In an optional embodiment, adjusting the recognition threshold of multiple first controls in the third user interface according to the light intensity range to which the light intensity data belongs includes: when the light intensity range is the weak light range, determining the recognition threshold according to the following first formula. : Where L is the light intensity data, The coefficient of the first-order term in the first formula is... The constant term of the first formula; when the light intensity range is the medium light range, the recognition threshold is determined according to the following second formula. : ,in, The first-order coefficient of the second formula is... The constant term in the second formula; when the light intensity range is the strong light range, the recognition threshold is determined according to the following third formula. : ,in, The first-order coefficient of the third formula is... is the constant term in the third formula.

[0008] In an optional embodiment, after adjusting the first user interface according to the cause of the anomaly, the method further includes: continuously collecting the rendering data stream of the third user interface, wherein the third user interface is the adjusted first user interface; determining whether the first byte value in the first data stream at the current moment is the same as the second byte value in the second data stream at the previous moment, wherein the rendering data stream includes the first data stream and the second data stream, and the byte positions of the first byte value and the second byte value are the same; if it is determined that the first byte value and the second byte value are not the same, loading a second control in the third user interface according to a first byte sequence composed of multiple first byte values, wherein the second control is a control other than the multiple first controls in the third user interface.

[0009] In an optional embodiment, loading a second control in the third user interface based on a first byte sequence composed of multiple first byte values ​​includes: retrieving second byte sequences corresponding to multiple third controls in a control mapping library based on the first byte sequence; determining whether there is a third byte sequence identical to the first byte sequence among the multiple second byte sequences, wherein the multiple third controls correspond one-to-one with the multiple second byte sequences; if the existence of the third byte sequence is determined among the multiple second byte sequences, identifying the third control corresponding to the third byte sequence as the second control, and loading the second control in the third user interface based on the first position coordinates of the third control.

[0010] In an optional embodiment, after loading a second control in the third user interface based on a first byte sequence consisting of multiple first byte values, the method further includes: determining a fourth control based on the second position coordinates when a click operation of a target object on a second position coordinate is detected; determining an operation instruction of the target object based on the functional attributes of the fourth control; and determining a second terminal device based on the device identification information of the fourth control; and sending the operation instruction to the second terminal device to instruct the second terminal device to execute the operation instruction.

[0011] In an optional embodiment, calculating the mean square error (MSE) of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface includes: determining the MSE according to the following formula: ,in, Let be the pixel value of the i-th pixel among the plurality of first pixels. Let be the pixel value of the i-th pixel among the plurality of second pixels, and n be the number of the plurality of first pixels, wherein the number of the plurality of first pixels is the same as the number of the plurality of second pixels.

[0012] According to another aspect of the present invention, a user interface adjustment device is also provided, comprising: a construction module for constructing a second user interface conforming to an expected standard based on image data features in an actual first user interface; a calculation module for calculating the mean square error of a plurality of pixels based on a plurality of first pixels in the first user interface and a plurality of second pixels in the second user interface, wherein the plurality of pixels includes the plurality of first pixels and the plurality of second pixels; and an adjustment module for determining that the first user interface is abnormal when the mean square error is greater than a preset threshold, and adjusting the first user interface according to the cause of the abnormality.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described user interface adjustment method when running.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the user interface adjustment method through the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method for adjusting the user interface.

[0016] In this embodiment, a second user interface conforming to expected standards is constructed based on image data features in the actual first user interface; the mean square error of multiple pixels is calculated based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include multiple first pixels and multiple second pixels; if the mean square error is greater than a preset threshold, it is determined that the first user interface has an anomaly, and the first user interface is adjusted according to the cause of the anomaly. By adopting the above technical solution, the problem in related technologies that traditional UI anomaly detection schemes rely on simple rules and limited templates to identify UI rendering anomalies, resulting in low accuracy in anomaly detection is solved. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment for a user interface adjustment method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a method for adjusting the user interface according to an embodiment of this application;

[0021] Figure 3 This is a first schematic diagram of a method for adjusting the user interface according to an optional embodiment of this application;

[0022] Figure 4 This is a second schematic diagram of a user interface adjustment method according to an optional embodiment of this application;

[0023] Figure 5 This is a third schematic diagram of a method for adjusting the user interface according to an optional embodiment of this application;

[0024] Figure 6 This is a flowchart illustrating a method for adjusting the user interface according to an optional embodiment of this application;

[0025] Figure 7 This is a structural block diagram of a user interface adjustment device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to one aspect of the embodiments of this application, a method for adjusting a user interface is provided. This user interface adjustment method is widely applicable to digital control scenarios such as smart homes, smart home ecosystems, intelligence house ecosystems, whole-house intelligence, digital homes, networked homes, automated homes, electronic homes, digital homes, and digital families. It relies on technologies such as the Internet of Things (IoT), cloud computing, artificial intelligence (AI), digital technology, digital twins, and AR / VR to construct an intelligent, automated, personalized, and customized digital ecosystem. Optionally, in this embodiment, the above-mentioned user interface adjustment method can be applied to... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0029] The aforementioned network communication architecture includes, but is not limited to, two types of links: wired transmission and wireless connection. Among them, the wired network can cover wide area networks, metropolitan area networks, and local area networks, while the wireless network supports communication standards such as WIFI, Bluetooth, and 5G. It can adapt to the digital technology transmission needs of the Internet of Things (IoT), the Internet of Things, and the Internet of Things blockchain, ensuring stable interaction of digital information, enabling devices to make autonomous, accurate, precise, and accurate judgments and adjustments, and supporting personalized and customized intelligent capabilities such as voice interaction, remote control, and self-control. Terminal devices 102 include, but are not limited to, PCs, mobile phones, tablets, and all categories of digital home appliances, digital electrical appliances, smart home appliances, and whole-house smart devices. Specifically, they include: smart air conditioners, air conditioning equipment, smart range hoods, smart refrigerators, smart ovens, grilling and cooking equipment, smart stoves, smart washing machines, washing, rinsing, drying and garment processing equipment, smart water heaters, smart washing equipment, smart dishwashers, smart projectors, smart TVs, smart clothes racks, smart curtains, smart audio-visual equipment, smart sockets, smart speakers, smart speakers, smart fresh air systems, smart kitchen and bathroom systems, kitchens, cabinets, kitchen cabinet equipment, smart bathroom equipment, robot vacuum cleaners, robot window cleaners, robot mopping, air purifiers, smart steam ovens, smart microwave ovens, smart water heaters, smart air purifiers, smart water dispensers, smart door locks, smart lighting fixtures, smart light bulbs, smart air detectors, smart remote controls, and smart gateways. They also cover smart terminals for security, lighting, and other digital home furnishings.

[0030] This embodiment provides a method for adjusting the user interface. Figure 2 This is a flowchart of a user interface adjustment method according to an embodiment of this application, the process including the following steps:

[0031] Step S202: Construct a second user interface that meets the expected standards based on the image data features in the actual first user interface;

[0032] Step S204: Calculate the mean square error of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include the multiple first pixels and the multiple second pixels;

[0033] Step S206: If the mean square error is greater than a preset threshold, determine that the first user interface is abnormal, and adjust the first user interface according to the cause of the abnormality.

[0034] In this embodiment, a second user interface conforming to expected standards is constructed based on image data features in the actual first user interface; the mean square error of multiple pixels is calculated based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include multiple first pixels and multiple second pixels; if the mean square error is greater than a preset threshold, it is determined that the first user interface has an anomaly, and the first user interface is adjusted according to the cause of the anomaly. By adopting the above technical solution, the problem in related technologies that traditional UI anomaly detection schemes rely on simple rules and limited templates to identify UI rendering anomalies, resulting in low accuracy in anomaly detection is solved.

[0035] like Figure 3 As shown, the original image data (UI data) of the first user interface currently rendered on the first terminal device (equivalent to a cross-brand device) is obtained. This image data includes the pixel color, position, size, and texture information of all visible controls on the screen. A dynamic UI anomaly detection algorithm based on a diffusion model is used to determine whether there are any anomalies in the actual UI interface. This algorithm includes a Stable Diffusion reconstruction module, a pixel-level difference analysis module, and an anomaly judgment module. Specifically, the image data is input into a pre-trained Stable Diffusion reconstruction module. This module, based on a generative model trained on a large number of standard UI samples, uses a back-diffusion process to gradually denoise the Gaussian noise latent space, reconstructing a second user interface that conforms to the expected interaction specifications and design standards. This interface represents an ideal state interface under conditions of no anomalies and no rendering errors. Its structure and semantics are consistent with the first user interface, but its pixel-level performance conforms to the brand's UI design specifications and system rendering consistency standards. After reconstruction, the first user interface and the second user interface are aligned at the pixel level using a pixel-level difference analysis module. All valid pixels in corresponding areas of the two interfaces are selected, and their mean squared error (MSE) is calculated using the following formula: ,in, Let i be the pixel value of the i-th pixel among multiple first pixels. Let be the pixel value of the i-th pixel among multiple second pixels, and n be the number of multiple first pixels, the same as the number of multiple second pixels. When the calculated MSE value is greater than the preset anomaly detection threshold T (preset threshold), the anomaly detection module determines that there is a significant visual anomaly in the first user interface. Further, a local difference heatmap analysis is performed on the abnormal area to identify the anomaly type, such as: control offset (pixel displacement caused by coordinate offset), color distortion (color difference exceeding the tolerance range), blurring or incompleteness (missing high-frequency textures), text misalignment or occlusion, etc. Based on the anomaly type, the corresponding repair strategy is invoked: if it is a color deviation, the display color configuration is automatically calibrated; if it is a control position offset, the layout parameters are reset or the interface is re-rendered; if it is a control missing due to resource loading failure, the background component is reloaded or the user is prompted to manually refresh. Finally, the anomaly diagnosis results and repair actions are recorded in the log and fed back to the user interface to prompt "The interface has been automatically repaired" or "Please restart the application to restore normal display," thereby achieving closed-loop control from anomaly detection to intelligent repair.

[0036] In an optional embodiment, after adjusting the first user interface according to the cause of the anomaly, the method further includes: acquiring light intensity data of the area where the first terminal device is located, wherein the first user interface is used to display the status of the first terminal device; determining that the light intensity data belongs to a weak light range when the light intensity data is less than a first preset value; determining that the light intensity data belongs to a medium light range when the light intensity data is greater than or equal to the first preset value and less than a second preset value; determining that the light intensity data belongs to a strong light range when the light intensity data is greater than or equal to the second preset value; adjusting the recognition threshold of a plurality of first controls in a third user interface according to the light intensity range to which the light intensity data belongs, wherein the recognition threshold is the minimum reference value for recognizing the plurality of first controls, the light intensity range includes: the weak light range, the medium light range, and the strong light range, and the third user interface is the adjusted first user interface; and adjusting the visibility of the plurality of first controls in the third user interface according to the recognition threshold.

[0037] In an optional embodiment, adjusting the recognition threshold of multiple first controls in the third user interface according to the light intensity range to which the light intensity data belongs includes: when the light intensity range is the weak light range, determining the recognition threshold according to the following first formula. : Where L is the light intensity data, The coefficient of the first-order term in the first formula is... The constant term of the first formula; when the light intensity range is the medium light range, the recognition threshold is determined according to the following second formula. : ,in, The first-order coefficient of the second formula is... The constant term in the second formula; when the light intensity range is the strong light range, the recognition threshold is determined according to the following third formula. : ,in, The first-order coefficient of the third formula is... is the constant term in the third formula.

[0038] like Figure 4 As shown, the illumination-adaptive AR interface fusion detection system includes: a light intensity sensor data acquisition module, a threshold dynamic adjustment module, and a UI recognition module. The light intensity sensor data acquisition module collects ambient light intensity data L (unit: lux) of the area where the first terminal device is located at a fixed frequency. This light intensity data characterizes the lighting intensity of the current environment. The first user interface is the original graphical interface of the first terminal device currently displaying its operating status, such as a smart home appliance control panel or a voice assistant interaction screen. Subsequently, the threshold dynamic adjustment module compares the collected light intensity data L with three preset interval division thresholds: when the light intensity data is less than the first preset value, the current environment is determined to be in a low-light interval. In this case, insufficient ambient light may cause UI controls to appear dim and have reduced contrast, easily leading to missed recognition; when the light intensity data is greater than or equal to the first preset value and less than the second preset value, the current environment is determined to be in a medium-light interval, i.e., normal indoor lighting conditions; when the light intensity data is greater than or equal to the second preset value, the current environment is determined to be in a strong-light interval, such as direct sunlight or high-brightness lighting scenes. In this case, screen reflections and glare may cause control edges to become blurred or misjudged as noise. Based on the above light intensity range determination results, the recognition thresholds of multiple first controls (such as switch buttons, sliders, icons, etc.) in the third user interface (i.e. the first user interface after abnormal adjustment) are adaptively adjusted. The recognition threshold is the minimum reference value of the control confidence output by the target detection algorithm (such as YOLO). Only when the detection confidence is higher than the recognition threshold is the control confirmed as a valid target.

[0039] Specifically, in the low-light region, the first linear function is used. Calculate the recognition threshold using a second linear function in the medium light range. Calculate the recognition threshold using a third linear function in the strong light region. Calculate the recognition threshold. The coefficients of the three linear functions mentioned above ( , , , , , All data were obtained by collecting no fewer than 500 sets of lighting-control recognition data in real home environments, combining them with manually labeled results, and then using the least squares regression fitting method to ensure robustness and generalization ability in actual deployment. Finally, the adjusted recognition threshold was dynamically injected into the UI recognition module of the AR interface. This module, based on a deep learning-based object detection algorithm, performs confidence screening and spatial positioning of multiple first controls in real-time based on the new threshold, thereby avoiding missed detections in low light and false detections in strong light. This ensures that controls can be accurately recognized during AR virtual-real interaction, allowing users to accurately operate UI controls under different lighting conditions.

[0040] It should be clarified that the light intensity data in this embodiment is not only used to characterize ambient light intensity, but also to characterize the special light intensity data of the simulated real environment in the control terminal (e.g., mobile APP), such as the light intensity data corresponding to the reflection mode and gradient mode in the mobile APP.

[0041] In an optional embodiment, after adjusting the first user interface according to the cause of the anomaly, the method further includes: continuously collecting the rendering data stream of the third user interface, wherein the third user interface is the adjusted first user interface; determining whether the first byte value in the first data stream at the current moment is the same as the second byte value in the second data stream at the previous moment, wherein the rendering data stream includes the first data stream and the second data stream, and the byte positions of the first byte value and the second byte value are the same; if it is determined that the first byte value and the second byte value are not the same, loading a second control in the third user interface according to a first byte sequence composed of multiple first byte values, wherein the second control is a control other than the multiple first controls in the third user interface.

[0042] like Figure 5 As shown, the real-time UI streaming differential analysis engine includes: a data stream comparison module, a dynamic control capture module, and an operation command matching module. The real-time UI streaming differential analysis engine continuously collects continuous frame interface data streams (rendering data streams) of the third user interface (the first user interface after abnormal adjustments) during the rendering process. This data stream is a raw byte sequence output by the device system's underlying graphics rendering engine, containing control layout, attributes, states, and pixel information. Its format is standardized UI rendering buffer data, with each time step corresponding to a complete frame of interface state. The first data stream at the current moment is extracted with a fixed sampling period, and the second data stream from the previous moment is cached synchronously. The two constitute a continuous frame sequence in time. Subsequently, the data stream comparison module performs a byte-by-byte alignment comparison between the first and second data streams: for each byte position i (from the first byte to the last byte of the data stream), the value d of the i-th byte in the first data stream is extracted. 1iThe byte value d at the corresponding position in the second data stream 2i And perform a binary comparison. If d is at a certain byte position i... 1i ≠d 2i If position i is marked as a difference point, continue scanning along the data flow direction to collect multiple consecutive difference points. Based on the first byte value of multiple difference points, a first byte sequence is formed. The dynamic control capture module identifies and captures the newly dynamically loaded second control based on the first byte sequence and loads the second control into the third user interface.

[0043] In an optional embodiment, loading a second control in the third user interface based on a first byte sequence composed of multiple first byte values ​​includes: retrieving second byte sequences corresponding to multiple third controls in a control mapping library based on the first byte sequence; determining whether there is a third byte sequence identical to the first byte sequence among the multiple second byte sequences, wherein the multiple third controls correspond one-to-one with the multiple second byte sequences; if the existence of the third byte sequence is determined among the multiple second byte sequences, identifying the third control corresponding to the third byte sequence as the second control, and loading the second control in the third user interface based on the first position coordinates of the third control.

[0044] like Figure 5As shown, the dynamic control capture module uses the first byte sequence as the query key to sequentially traverse all second byte sequences in the control mapping library to determine whether there exists a third byte sequence that is completely identical to the first byte sequence (i.e., the byte length, order, and value are all the same). The control mapping library pre-stores multiple third controls (such as pop-up menus, dynamically loaded buttons, video playback controls injected into WebView, temporary status prompts, etc.) and their unique corresponding second byte sequences in the rendering data stream. Each second byte sequence consists of several byte offsets and their expected values ​​(for example, the second byte sequence of the "play button" control might be: [0x5A, 0x23, 0x8F, 0x01, 0x00, 0x08], corresponding to fields such as control name, type code, and coordinate offset). When a third byte sequence that perfectly matches the first byte sequence is successfully retrieved in the control mapping library, the third control bound to this third byte sequence is determined to be the currently loading second control. At this point, its first position coordinates are extracted from the metadata of the third control. These coordinates represent the theoretical position under standard resolution and layout, and have been pre-collected and stored in the control mapping library during the control registration phase using UI automation tools. Based on the real-time scaling ratio, screen resolution, and coordinate system offset of the current third user interface, the first position coordinates are dynamically adapted and calculated to generate actual rendering coordinates suitable for the current display environment. The second control is then loaded and drawn in the third user interface, making it visible and operable on AR or interactive interfaces. It should be noted that the metadata of the control includes, but is not limited to, the control's coordinates, control type (such as "button" or "slider"), bound function instructions (such as "SET_VOLUME"), and the device identifier (such as "smart_tv_07").

[0045] In an optional embodiment, after loading a second control in the third user interface based on a first byte sequence consisting of multiple first byte values, the method further includes: determining a fourth control based on the second position coordinates when a click operation of a target object on a second position coordinate is detected; determining an operation instruction of the target object based on the functional attributes of the fourth control; and determining a second terminal device based on the device identification information of the fourth control; and sending the operation instruction to the second terminal device to instruct the second terminal device to execute the operation instruction.

[0046] like Figure 5As shown, in a unified interaction scenario across smart devices from different brands, when a target object (such as a user's finger, stylus, or AR cursor) performs a click operation on a second position coordinate in the third user interface, the second position coordinate is input into a real-time updated control mapping table through the operation instruction matching module. This mapping table is dynamically maintained by the aforementioned real-time UI streaming differential analysis engine, which records the precise boundary areas (top-left corner coordinates, width, and height) of all identified controls (including static first controls and dynamically loaded second controls) in the current third user interface in the screen coordinate system, along with their corresponding unique control identifiers. A spatial range judgment algorithm determines whether the second position coordinate falls within the response area of ​​a control. If it does, the control associated with that control identifier is defined as the fourth control. This fourth control is the interaction target intended by the user; it may be a pre-set fixed control or a second control dynamically loaded through byte sequence matching. Both have equal processing permissions in the mapping table. Subsequently, based on the functional attribute information bound to the fourth control, preset standard operation instructions are extracted from its metadata. These functional attributes are stored in the control mapping library as structured fields, for example, in the format: {"cmd": "SET_TEMPERATURE", "param": 24, "protocol": "MQTT"}. Simultaneously, the device identification information associated with the fourth control is read, such as the device UUID, MAC address, distributed device ID, or cloud platform device code. Based on this device identification information, the second terminal device (such as a smart air conditioner, smart TV, or third-party brand air purifier) ​​corresponding to the fourth control is located. After completing control recognition, instruction extraction, and target device location, the standard operation instructions are encapsulated into data packets using a preset cross-device communication protocol and sent to the second terminal device. At the receiving end, after processing by the protocol parsing, permission verification, and execution scheduling modules, the instruction drives the corresponding hardware function to execute, such as adjusting the temperature, changing channels, or starting a cleaning mode.

[0047] In an optional embodiment, calculating the mean square error (MSE) of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface includes: determining the MSE according to the following formula: ,in, Let be the pixel value of the i-th pixel among the plurality of first pixels. Let be the pixel value of the i-th pixel among the plurality of second pixels, and n be the number of the plurality of first pixels, wherein the number of the plurality of first pixels is the same as the number of the plurality of second pixels.

[0048] To better understand the process of adjusting the user interface described above, the following description, in conjunction with optional embodiments, further illustrates the implementation flow of the user interface adjustment method, but is not intended to limit the technical solutions of the embodiments of this application.

[0049] Figure 6 This is a flowchart illustrating a method for adjusting the user interface according to an optional embodiment of the present invention, such as... Figure 6 As shown, it specifically includes the following:

[0050] The UI data of the first user interface currently rendered on the cross-brand device (first terminal device) is transmitted to a dynamic UI anomaly detection algorithm based on a diffusion model. This algorithm then inputs the UI data into a pre-trained Stable Diffusion reconstruction module. This module, based on a generative model trained on a massive number of standard UI samples, gradually denoises the Gaussian noise latent space through a back-diffusion process, reconstructing a second user interface that conforms to the expected interaction specifications and design standards. After reconstruction, a pixel-level difference analysis module performs pixel-level alignment processing on the first and second user interfaces, selecting all effective pixels in corresponding areas of the two interfaces and calculating their mean square error. ,in, Let be the pixel value of the i-th pixel among multiple first pixels in the first user interface. Let be the pixel value of the i-th pixel among multiple second pixels in the second user interface, and n be the number of multiple first pixels, which is the same as the number of multiple second pixels. When the calculated MSE value is greater than the preset anomaly judgment threshold T (preset threshold), the anomaly judgment module determines that there is a significant visual anomaly in the first user interface, returns the anomaly result to the cross-brand device, and the cross-brand device adjusts the first user interface according to the cause of the anomaly to obtain the third user interface.

[0051] Furthermore, cross-brand devices send continuous frame UI rendering data streams of the third user interface to the real-time UI streaming differential analysis engine. The real-time UI streaming differential analysis engine extracts the first data stream of the current moment from the continuous frame UI rendering data stream and synchronously caches the second data stream of the previous moment. Subsequently, the first data stream and the second data stream are compared byte by byte through the data stream comparison module: for each byte position i (from the first byte to the last byte of the data stream), the value d of the i-th byte in the first data stream is extracted. 1i The byte value d at the corresponding position in the second data stream 2i And perform a binary comparison. If d is at a certain byte position i... 1i ≠d 2iIf position i is marked as a difference point, the scanning continues along the data flow direction, collecting multiple consecutive difference points. A first byte sequence is formed based on the first byte values ​​of these multiple difference points. The dynamic control capture module identifies and captures the newly dynamically loaded second control based on this first byte sequence and provides the control information of the second control to the operation instruction matching module. Upon receiving a click operation on the UI interface from the target object, the operation instruction matching module determines the corresponding dynamic control and the target object's operation instruction based on the click's coordinates, matches the operation instruction with the dynamic control, and feeds back the matching result to the cross-brand device.

[0052] On the other hand, the illumination-adaptive AR interface fusion detection system receives ambient light intensity data L collected at a fixed frequency by a light intensity sensor data acquisition module. This light intensity data is then input to a threshold dynamic adjustment module, which compares the light intensity data L with three preset interval thresholds: when the light intensity data is less than a first preset value, the current environment is determined to be in a low-light interval; when the light intensity data is greater than or equal to the first preset value and less than a second preset value, the current environment is determined to be in a medium-light interval; and when the light intensity data is greater than or equal to the second preset value, the current environment is determined to be in a high-light interval. In the low-light interval, a first linear function is used. Calculate the recognition threshold using a second linear function in the medium light range. Calculate the recognition threshold using a third linear function in the strong light region. The recognition threshold is calculated. The adjusted recognition threshold is dynamically injected into the UI recognition module of the AR interface. The UI recognition module performs confidence filtering and spatial positioning on multiple primary controls in real time based on the new threshold, and sends the UI recognition results to cross-brand devices.

[0053] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, operating terminal, or network device, etc.) to execute the methods of the various embodiments of this application.

[0054] Figure 7 This is a structural block diagram of a user interface adjustment device according to an embodiment of this application; as shown... Figure 7 As shown, it includes:

[0055] Module 72 is used to construct a second user interface that conforms to the expected standards based on the image data features in the actual first user interface;

[0056] The calculation module 74 is used to calculate the mean square error of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include the multiple first pixels and the multiple second pixels;

[0057] The adjustment module 76 is used to determine that the first user interface is abnormal when the mean square error is greater than a preset threshold, and to adjust the first user interface according to the cause of the abnormality.

[0058] In this embodiment, a second user interface conforming to expected standards is constructed based on image data features in the actual first user interface; the mean square error of multiple pixels is calculated based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include multiple first pixels and multiple second pixels; if the mean square error is greater than a preset threshold, it is determined that the first user interface has an anomaly, and the first user interface is adjusted according to the cause of the anomaly. By adopting the above technical solution, the problem in related technologies that traditional UI anomaly detection schemes rely on simple rules and limited templates to identify UI rendering anomalies, resulting in low accuracy in anomaly detection is solved.

[0059] In an optional embodiment, the adjustment module 76 is further configured to acquire light intensity data of the area where the first terminal device is located, wherein the first user interface is used to display the status of the first terminal device; when the light intensity data is less than a first preset value, determine that the light intensity data belongs to a weak light range; when the light intensity data is greater than or equal to the first preset value and less than a second preset value, determine that the light intensity data belongs to a medium light range; when the light intensity data is greater than or equal to the second preset value, determine that the light intensity data belongs to a strong light range; adjust the recognition threshold of a plurality of first controls in the third user interface according to the light intensity range to which the light intensity data belongs, wherein the recognition threshold is the minimum reference value for recognizing the plurality of first controls, the light intensity range includes: the weak light range, the medium light range, and the strong light range, and the third user interface is the adjusted first user interface; and adjust the visibility of the plurality of first controls in the third user interface according to the recognition threshold.

[0060] In an optional embodiment, the adjustment module 76 is further configured to determine the identification threshold according to the following first formula when the light intensity range is the weak light range. : Where L is the light intensity data, The coefficient of the first-order term in the first formula is... The constant term of the first formula; when the light intensity range is the medium light range, the recognition threshold is determined according to the following second formula. : ,in, The first-order coefficient of the second formula is... The constant term in the second formula; when the light intensity range is the strong light range, the recognition threshold is determined according to the following third formula. : ,in, The first-order coefficient of the third formula is... is the constant term in the third formula.

[0061] In an optional embodiment, the adjustment module 76 is further configured to continuously collect the rendering data stream of the third user interface, wherein the third user interface is the adjusted first user interface; determine whether the first byte value in the first data stream at the current moment is the same as the second byte value in the second data stream at the previous moment, wherein the rendering data stream includes the first data stream and the second data stream, and the byte positions of the first byte value and the second byte value are the same; if it is determined that the first byte value and the second byte value are not the same, load a second control in the third user interface according to a first byte sequence composed of multiple first byte values, wherein the second control is a control other than the multiple first controls in the third user interface.

[0062] In an optional embodiment, the adjustment module 76 is further configured to retrieve second byte sequences corresponding to multiple third controls in the control mapping library based on the first byte sequence, determine whether there is a third byte sequence in the multiple second byte sequences that is the same as the first byte sequence, wherein the multiple third controls correspond one-to-one with the multiple second byte sequences; if it is determined that the third byte sequence exists in the multiple second byte sequences, the third control corresponding to the third byte sequence is determined as the second control, and the second control is loaded in the third user interface according to the first position coordinates of the third control.

[0063] In an optional embodiment, the adjustment module 76 is further configured to, upon detecting a click operation of the target object on the second position coordinates, determine a fourth control based on the second position coordinates; determine the operation instruction of the target object based on the functional attributes of the fourth control; and determine a second terminal device based on the device identification information of the fourth control; and send the operation instruction to the second terminal device to instruct the second terminal device to execute the operation instruction.

[0064] In an optional embodiment, the calculation module 74 is further configured to determine the mean square error (MSE) according to the following formula: ,in, Let be the pixel value of the i-th pixel among the plurality of first pixels. Let be the pixel value of the i-th pixel among the plurality of second pixels, and n be the number of the plurality of first pixels, wherein the number of the plurality of first pixels is the same as the number of the plurality of second pixels.

[0065] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0066] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0067] S1, construct a second user interface that meets the expected standards based on the image data features in the actual first user interface;

[0068] S2, calculate the mean square error of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include the multiple first pixels and the multiple second pixels;

[0069] S3, if the mean square error is greater than a preset threshold, determine that the first user interface is abnormal, and adjust the first user interface according to the cause of the abnormality.

[0070] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0071] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0072] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0073] S1, construct a second user interface that meets the expected standards based on the image data features in the actual first user interface;

[0074] S2, calculate the mean square error of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include the multiple first pixels and the multiple second pixels;

[0075] S3, if the mean square error is greater than a preset threshold, determine that the first user interface is abnormal, and adjust the first user interface according to the cause of the abnormality.

[0076] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0077] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0078] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0079] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0080] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0081] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0082] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for adjusting a user interface, characterized in that, include: Construct a second user interface that meets the expected standards based on the image data features in the actual first user interface; The mean square error of multiple pixels is calculated based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include the multiple first pixels and the multiple second pixels; If the mean square error is greater than a preset threshold, it is determined that the first user interface is abnormal, and the first user interface is adjusted according to the cause of the abnormality.

2. The user interface adjustment method according to claim 1, characterized in that, After adjusting the first user interface according to the cause of the anomaly, the method further includes: Acquire light intensity data of the area where the first terminal device is located, wherein the first user interface is used to display the status of the first terminal device; If the light intensity data is less than a first preset value, it is determined that the light intensity data belongs to the weak light range; If the light intensity data is greater than or equal to the first preset value and the light intensity data is less than the second preset value, the light intensity data is determined to belong to the medium light range. If the light intensity data is greater than or equal to the second preset value, it is determined that the light intensity data belongs to the strong light range; The recognition thresholds of multiple first controls in the third user interface are adjusted according to the light intensity range to which the light intensity data belongs. The recognition threshold is the minimum reference value for recognizing the multiple first controls. The light intensity range includes: the weak light range, the medium light range, and the strong light range. The third user interface is the adjusted first user interface. The visibility of the plurality of first controls in the third user interface is adjusted according to the recognition threshold.

3. The user interface adjustment method according to claim 2, characterized in that, Adjusting the recognition thresholds of multiple first controls in the third user interface according to the light intensity range to which the light intensity data belongs, including: When the light intensity range is the weak light range, the recognition threshold is determined according to the following first formula. : Where L is the light intensity data, The coefficient of the first-order term in the first formula is... This is the constant term in the first formula; When the light intensity range is the medium light range, the recognition threshold is determined according to the following second formula. : ,in, The first-order coefficient of the second formula is... This is the constant term in the second formula; When the light intensity range is the strong light range, the recognition threshold is determined according to the following third formula. : ,in, The first-order coefficient of the third formula is... is the constant term in the third formula.

4. The user interface adjustment method according to claim 1, characterized in that, After adjusting the first user interface according to the cause of the anomaly, the method further includes: Continuously collect the rendering data stream of the third user interface, wherein the third user interface is the adjusted first user interface; Determine whether the first byte value in the first data stream at the current moment is the same as the second byte value in the second data stream at the previous moment, wherein the rendering data stream includes the first data stream and the second data stream, and the byte positions of the first byte value and the second byte value are the same; If it is determined that the first byte value and the second byte value are different, a second control is loaded in the third user interface according to a first byte sequence composed of multiple first byte values, wherein the second control is a control other than the multiple first controls in the third user interface.

5. The user interface adjustment method according to claim 4, characterized in that, Loading a second control in the third user interface based on a first byte sequence consisting of multiple first byte values ​​includes: Based on the first byte sequence, retrieve the second byte sequences corresponding to multiple third controls in the control mapping library, and determine whether there is a third byte sequence in the multiple second byte sequences that is the same as the first byte sequence, wherein the multiple third controls correspond one-to-one with the multiple second byte sequences; If the third byte sequence is found to exist among multiple second byte sequences, the third control corresponding to the third byte sequence is determined as the second control, and the second control is loaded in the third user interface according to the first position coordinates of the third control.

6. The user interface adjustment method according to claim 4, characterized in that, After loading the second control in the third user interface based on a first byte sequence consisting of multiple first byte values, the method further includes: If a click operation on the second position coordinate is detected by the target object, the fourth control is determined based on the second position coordinate; The operation instructions for the target object are determined based on the functional attributes of the fourth control, and the second terminal device is determined based on the device identification information of the fourth control; The operation instruction is sent to the second terminal device to instruct the second terminal device to execute the operation instruction.

7. The method for adjusting the user interface according to claim 1, characterized in that, Calculating the mean square error of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface includes: The mean square error (MSE) is determined according to the following formula: ,in, Let be the pixel value of the i-th pixel among the plurality of first pixels. Let be the pixel value of the i-th pixel among the plurality of second pixels, and n be the number of the plurality of first pixels, wherein the number of the plurality of first pixels is the same as the number of the plurality of second pixels.

8. A user interface adjustment device, characterized in that, include: A building module is used to construct a second user interface that conforms to expected standards based on the image data features in the actual first user interface. A calculation module is configured to calculate the mean square error of multiple pixels based on multiple first pixels in the first user interface and multiple second pixels in the second user interface, wherein the multiple pixels include the multiple first pixels and the multiple second pixels; The adjustment module is used to determine that the first user interface is abnormal when the mean square error is greater than a preset threshold, and to adjust the first user interface according to the cause of the abnormality.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the user interface adjustment method as described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the user interface adjustment method according to any one of claims 1 to 7 through the computer program.

Images