Interface processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an interface processing method and device, electronic equipment and a storage medium, and relates to the technical field of graphical interface processing.The method comprises the steps that a first image corresponding to a first graphical user interface and a second image corresponding to a second graphical user interface are obtained, the first image and the second image are images obtained by triggering screen capture at the same moment; extracting first image information from the first image, extracting second image information from the second image, and performing similarity matching according to the first image information and the second image information to obtain an interface difference degree between the first image and the second image; and according to the interface difference degree between the mobile terminal and each central control screen, outputting the interface detection report for the central control screen, thereby simplifying the interface consistency test process, improving the test efficiency, and realizing batch test.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of graphical interface processing, in particular to an interface processing method, an interface processing device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the popularity of smart home, the central control screen and mobile terminal (such as application program, etc.) become important interactive tools for users to control smart home devices (such as air conditioner, fan, curtain, etc.). In order to ensure user experience, the consistency of the content displayed between the central control screen and the mobile terminal needs to be ensured, such as the running state of the air conditioner displayed on the central control screen and the mobile terminal needs to be consistent.

[0003] For the processing of the interface, it often depends on manual operation and naked eye comparison, which makes the test method inefficient, easy to be affected by human error, and difficult to adapt to multi-device and large-scale test scenarios. SUMMARY

[0004] The embodiments of the present application provide an interface processing method, device, electronic device and computer readable storage medium to solve or partially solve the problems of low test efficiency, large error and difficulty in batch testing in the process of interface consistency testing.

[0005] The embodiments of the present application disclose an interface processing method, which relates to at least one mobile terminal and at least two central control screens, the mobile terminal and the central control screen are connected to the same smart home device; the mobile terminal provides a first graphical user interface, and the central control screen provides a second graphical user interface, and the method comprises:

[0006] Obtaining a first image corresponding to the first graphical user interface and a second image corresponding to the second graphical user interface, the first image and the second image are images obtained by triggering screen capture at the same time;

[0007] Extracting first image information from the first image and second image information from the second image, and performing similarity matching according to the first image information and the second image information to obtain the interface difference degree between the first image and the second image;

[0008] According to the interface difference degree between the mobile terminal and each central control screen, an interface detection report for the central control screen is output.

[0009] In some possible implementation manners, the first image information at least includes first text information and first icon information, the second image information at least includes second text information and second icon information, and the similarity matching based on the first image information and the second image information to obtain the interface difference degree between the first image and the second image includes:

[0010] matching the first text information with the second text information to determine a text similarity;

[0011] matching the first icon information with the second icon information to determine an icon similarity;

[0012] calculating based on the text similarity and the icon similarity to obtain the interface difference degree between the first image and the second image.

[0013] In some possible implementation manners, the matching the first text information with the second text information to determine a text similarity includes:

[0014] matching a first character in the first text information and the second text information, and calculating an edit distance of the second text information converted into the first text information in a matching process;

[0015] obtaining a first text length of the first text information and a second text length of the second text information, and converting based on the first text length, the second text length and the edit distance to obtain the text similarity between the first image and the second image.

[0016] In some possible implementation manners, the first icon information includes a first icon region, and the second icon information includes a second icon region, and the matching the first icon information with the second icon information to determine an icon similarity includes:

[0017] obtaining a first pixel value of each first pixel point in the first icon region;

[0018] obtaining a second pixel value of each second pixel point in the second icon region;

[0019] obtaining a stable constant for the first icon region and the second icon region;

[0020] calculating based on the first pixel value, the second pixel value and the stable constant to obtain the icon similarity between the first image and the second image.

[0021] In some possible implementation manners, the calculating, according to the first pixel value, the second pixel value and the stability constant, to obtain the icon similarity between the first image and the second image comprises the following steps.

[0022] calculating a first average value corresponding to the first pixel value and a first variance;

[0023] calculating a second average value corresponding to the second pixel value and a second variance;

[0024] calculating a covariance between the first variance and the second variance;

[0025] calculating, by using the first average value, the second average value, the first variance, the second variance, the covariance and the stability constant, to obtain the icon similarity between the first image and the second image.

[0026] In some possible implementation manners, the calculating, based on the text similarity and the icon similarity, to obtain the interface difference between the first image and the second image comprises the following steps.

[0027] obtaining a first weight value corresponding to the text similarity;

[0028] obtaining a second weight value corresponding to the icon similarity;

[0029] calculating, by using the text similarity, the first weight value, the icon similarity and the second weight value, to obtain the interface difference between the first image and the second image.

[0030] In some possible implementation manners, the outputting, according to the interface difference between the mobile terminal and each of the central control screens, of the interface detection report for the central control screens comprises the following steps.

[0031] calculating an average difference between each of the interface differences;

[0032] calculating a standard deviation between each of the interface differences and the average difference;

[0033] calculating, by using the average difference and the standard deviation, to obtain a target detection value;

[0034] regarding a central control screen with an interface difference greater than or equal to the target detection value as an abnormal device, and generating an abnormal device mark corresponding to the abnormal device;

[0035] outputting, based on one of the interface difference and the abnormal device mark, the interface detection report for the central control screens.

[0036] The embodiment of the present application also discloses a processing device of an interface, relating to at least one mobile terminal and at least two central control screens, the mobile terminal and the central control screens being connected to the same smart home device; the mobile terminal provides a first graphical user interface, the central control screens provide a second graphical user interface, and the device comprises:

[0037] an image acquisition module, configured to acquire a first image corresponding to the first graphical user interface and a second image corresponding to the second graphical user interface, the first image and the second image being images obtained by triggering a screenshot at the same time;

[0038] a test module, configured to extract first image information from the first image, extract second image information from the second image, and perform similarity matching according to the first image information and the second image information to obtain an interface difference degree between the first image and the second image;

[0039] an output module, configured to output an interface detection report for the central control screens according to the interface difference degree between the mobile terminal and each of the central control screens.

[0040] In some possible implementation manners, the first image information at least includes first text information and first icon information, the second image information at least includes second text information and second icon information, and the test module is specifically configured to:

[0041] match the first text information with the second text information to determine a text similarity;

[0042] match the first icon information with the second icon information to determine an icon similarity;

[0043] calculate the text similarity and the icon similarity to obtain the interface difference degree between the first image and the second image.

[0044] In some possible implementation manners, the test module is specifically configured to:

[0045] match the first character in the first text information with the first character in the second text information, and calculate an edit distance of the second text information converted into the first text information in a matching process;

[0046] acquire a first text length of the first text information and a second text length of the second text information, and convert the first text length, the second text length and the edit distance to obtain a text similarity between the first image and the second image.

[0047] In some possible implementation manners, the first icon information includes a first icon region, and the second icon information includes a second icon region, and the test module is specifically configured to:

[0048] obtain a first pixel value of each first pixel point in the first icon region;

[0049] obtain a second pixel value of each second pixel point in the second icon region;

[0050] obtain a stable constant for the first icon region and the second icon region;

[0051] perform calculation according to the first pixel value, the second pixel value, and the stable constant to obtain an icon similarity between the first image and the second image.

[0052] In some possible implementation manners, the test module is specifically configured to:

[0053] calculate a first average value and a first variance corresponding to the first pixel value;

[0054] calculate a second average value and a second variance corresponding to the second pixel value;

[0055] calculate a covariance between the first variance and the second variance;

[0056] perform calculation by using the first average value, the second average value, the first variance, the second variance, the covariance, and the stable constant to obtain the icon similarity between the first image and the second image.

[0057] In some possible implementation manners, the test module is specifically configured to:

[0058] obtain a first weight value corresponding to the text similarity;

[0059] obtain a second weight value corresponding to the icon similarity;

[0060] perform calculation by using the text similarity, the first weight value, the icon similarity, and the second weight value to obtain an interface difference degree between the first image and the second image.

[0061] In some possible implementation manners, the output module is specifically configured to:

[0062] calculate an average difference degree between each interface difference degree;

[0063] calculate a standard deviation between each interface difference degree and the average difference degree;

[0064] The average difference degree and the standard deviation are used for calculation to obtain a target detection value;

[0065] A central control screen with an interface difference degree greater than or equal to the target detection value is regarded as an abnormal device, and an abnormal device mark corresponding to the abnormal device is generated;

[0066] An interface detection report for the central control screen is output based on one of the interface difference degree and the abnormal device mark.

[0067] An electronic device is also disclosed in the embodiments of the present application, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;

[0068] The memory is used to store a computer program.

[0069] The processor is used to execute the program stored on the memory, and realize the method as described in the embodiments of the present application.

[0070] A computer readable storage medium is also disclosed in the embodiments of the present application, which stores instructions, and when executed by one or more processors, causes the processor to execute the method as described in the embodiments of the present application.

[0071] The embodiments of the present application have the following advantages:

[0072] In the embodiments of the present application, in the process of detecting the consistency of interface display between the central control screen and the mobile terminal, the central control screen and the mobile terminal can be connected to the same smart home device to ensure the consistency of device control. The first image corresponding to the first graphical user interface and the second image corresponding to the second graphical user interface are obtained, the first image and the second image are images obtained by triggering screen capture at the same time, then the first image information is extracted from the first image, and the second image information is extracted from the second image, and the similarity between the first image information and the second image information is matched to obtain the interface difference degree between the first image and the second image. Then, based on the interface difference degree between the mobile terminal and each central control screen, an interface detection report for the central control screen is output, so that when detecting the interface consistency of different devices, the interface display content at the same time is extracted for image information, and the interface difference degree is determined based on the image information, which simplifies the process of interface consistency test, improves the test efficiency, and through matching the content displayed by at least one mobile terminal with the content displayed by at least two central control screens, batch testing can be realized, further improving the test efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1is a step flow chart of a processing method of an interface provided in an embodiment of the present application;

[0074] Figure 2 is a structural schematic diagram of device communication provided in an embodiment of the present application;

[0075] Figure 3 is a structural block diagram of a processing device of an interface provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the above objectives, characteristics and advantages of the present application more apparent, obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0077] As an example, in the process of controlling the smart home device, the user can control the smart home device through the control approaches provided by the terminal device, voice control, remote controller and the smart home device itself. Among them, the terminal device can include mobile terminal and central control screen, etc. In order to ensure the user's experience, it is necessary to ensure the consistency of the content displayed between the mobile terminal and the central control screen. However, in the process of detecting the interface consistency, it is often dependent on manual operation and naked eye comparison, so that the test method is inefficient, easy to be affected by human error, and difficult to adapt to multi-device, large-scale test scenarios, etc.

[0078] To this end, in the present application, in the process of detecting the interface display consistency between the central control screen and the mobile terminal, the central control screen and the mobile terminal can be connected to the same smart home device to ensure the consistency of device control. By obtaining a first image corresponding to a first graphical user interface and a second image corresponding to a second graphical user interface, the first image and the second image are images obtained by triggering the screen at the same time. Then, the first image information is extracted from the first image, and the second image information is extracted from the second image. According to the first image information and the second image information, the similarity is matched to obtain the interface difference degree between the first image and the second image. Then, according to the interface difference degree between the mobile terminal and each central control screen, the interface detection report for the central control screen is output, so that when detecting the interface consistency of different devices, for the interface display content at the same time, the corresponding image information is extracted, and the corresponding interface difference degree is determined based on the image information, which simplifies the process of interface consistency test, improves the test efficiency, and by matching the content displayed by at least one mobile terminal with the content displayed by at least two central control screens, batch testing can be realized, further improving the test efficiency.

[0079] REFERENCE Figure 1, a step flow chart of a processing method of an interface provided in an embodiment of the present application is shown, involving at least one mobile terminal and at least two central control screens, the mobile terminal and the central control screens being connected to the same smart home device; the mobile terminal provides a first graphical user interface, and the central control screen provides a second graphical user interface, which can specifically include the following steps:

[0080] In step 101, a first image corresponding to the first graphical user interface and a second image corresponding to the second graphical user interface are obtained, the first image and the second image being images triggered by screenshot at the same time;

[0081] In the embodiments of the present application, the smart home device can be an air conditioner, a television, a curtain, a fan, a sweeping robot, etc., and the user can remotely control the smart home device through a mobile terminal, a central control screen (used for centralized control and management of all networked smart home devices, and can provide a visual interactive interface), etc. The central control screen and the mobile terminal can communicate with the smart home device through wired or wireless means, so as to control the smart home device through the corresponding network.

[0082] It should be noted that a corresponding application program can be run in the mobile terminal, and a corresponding application interface can be presented through the graphical user interface provided by the mobile terminal. In the application interface, the device state, the interactive control, the text information, etc. corresponding to the smart home device can be presented.

[0083] In an example, the control interface displayed by the mobile terminal can be taken as a reference, and it can be detected whether the control interface displayed by the central control screen is consistent with the control interface displayed by the mobile terminal. Alternatively, the control interface displayed by the central control screen can be taken as a reference, and it can be detected whether the control interface displayed by the mobile terminal is consistent with the control interface displayed by the central control screen, etc. The present application does not limit this.

[0084] For example, the control interface displayed by the mobile terminal is taken as a reference for example description, with reference to Figure 2 , a structural schematic diagram of device communication provided in an embodiment of the present application is shown. In the process of interface consistency testing, one mobile terminal and multiple central control screens can be connected to the same smart home device. The first control interface corresponding to the smart home device can be displayed through the first graphical user interface provided by the mobile terminal, and the second control interface corresponding to the smart home device can be displayed through each central control screen, so as to determine whether the content displayed between the first control interface and the second control interface is consistent.

[0085] Optionally, during the testing, the mobile terminal, the central control screen and the like can be connected to a PC (Personal Computer) terminal, and the data generated during the testing of the mobile terminal, the central control screen and the like can be stored and analyzed through the PC terminal. For example, a corresponding storage directory can be created in the PC terminal, and the data can be named according to the ID information of the central control screen and the time stamp of the data generation, so as to better distinguish the corresponding data of different devices, different times and different test contents, and the present application does not limit this.

[0086] Correspondingly, during the testing, the tester can input the test instruction for the mobile terminal and the central control screen, so as to trigger the screenshot of the content displayed by the mobile terminal and the central control screen at the same time, obtain the first image corresponding to the first graphical user interface and the second image corresponding to the second graphical user interface, so as to obtain the images displayed by different devices at the same time, so as to analyze the interface consistency of different control devices connected to the same smart home device.

[0087] Step 102, extracting first image information from the first image and second image information from the second image, and performing similarity matching according to the first image information and the second image information to obtain the interface difference degree between the first image and the second image;

[0088] After obtaining the first image corresponding to the mobile terminal and the second image corresponding to each central control screen, the first image information can be extracted from the first image, and the second image information can be extracted from the second image, and the similarity matching can be performed according to the first image information and the second image information to obtain the interface difference degree between the first image and the second image. By extracting the corresponding image information and determining the corresponding interface difference degree based on the image information, the process of the interface consistency test is simplified, and the test efficiency is improved. And by matching the content displayed by at least one mobile terminal with the content displayed by at least two central control screens, batch testing can be realized, further improving the test efficiency. The image information can include text, icons, pixel information and the like contained in the image.

[0089] In some possible implementation manners, the first image information at least includes first text information and first icon information, and the second image information at least includes second text information and second icon information, wherein the text information can be text content contained in an image, and the icon information can be graphical content such as an interactive control contained in the image, which represents a function or a state through a graphical symbol. Then, the text similarity can be determined by matching the first text information with the second text information, the icon similarity can be determined by matching the first icon information with the second icon information, and the interface difference degree between the first image and the second image can be obtained based on calculation of the text similarity and the icon similarity, so that the difference between the text and the icon is fully considered, the interface consistency is tested from multiple dimensions, the difference between different interfaces is comprehensively and accurately captured, and the accuracy of the test is ensured.

[0090] The text similarity represents the difference degree between texts in content displayed by different devices, the icon similarity represents the difference degree between icons in content displayed by different devices, and accordingly, the interface difference degree represents the difference degree of content displayed by different devices as a whole, so that the consistency test can be performed in a targeted manner based on a single dimension, and the interface consistency test can be performed from multiple dimensions, the difference between different interfaces is comprehensively and accurately captured, and the accuracy of the test is ensured.

[0091] For the text similarity, the first text information can be matched with the second text information, the editing distance of the second text information converted into the first text information in a matching process can be calculated by matching the first character in the first text information and the second text information, the first text length of the first text information and the second text length of the second text information can be obtained, and the first text length, the second text length, and the editing distance can be converted to obtain the text similarity between the first image and the second image, so that the similarity between different text information displayed by the mobile terminal and the central control screen can be calculated, and the difference between the two can be analyzed from the text dimension.

[0092] In some examples, the editing distance between the first text information and the second text information can be calculated by the following formula, and the calculation result is converted into the corresponding text similarity after the corresponding result is calculated.

[0093]

[0094] Wherein, s1 represents the first string (i.e. the first text information); s2 represents the second string (i.e. the second text information); s1[0] represents the first character of s1 (i.e. the first character in the first text information); s2[0] represents the first character of s2 (i.e. the first character in the second text information); s1[1:] represents the substring of s1 starting from the second character (i.e. the part of the first text information after the first character); s2[1:] represents the substring of s2 starting from the second character (i.e. the part of the second text information after the first character); min{...} represents the minimum value function, taking the minimum value in the parentheses, used to select the optimal editing path; 1+min{...} represents the editing operation cost plus 1, indicating the minimum additional cost of one insertion, deletion or replacement operation, through which the minimum editing operation number required to convert s1 to s2 can be calculated, used to determine the similarity between different texts.

[0095] After obtaining the corresponding Levenshtein distance, it can be converted into a text similarity by the following formula:

[0096]

[0097] Wherein, Text Similarity represents the text similarity, i.e. the similarity percentage between two text information, with a value range of 0-1; Levenshtein(s1,s2) is the Levenshtein distance (i.e. the editing distance between s1 and s2) calculated in the above embodiment; len(s1) represents the text length of s1 (i.e. the number of characters in the first text information); len(s2) represents the text length of s2 (i.e. the number of characters in the second text information); max(len(s1),len(s2)) represents the larger value of the lengths of s1 and s2, which can be used for normalization to normalize the denominator to ensure similarity normalization); 1-... represents the similarity calculation, which is obtained by subtracting the normalized editing distance by 1, through which the Levenshtein distance can be normalized into a similarity value of 0-1, used to quantify the text consistency.

[0098] For example, assuming the first text information is "kitten" and the second text information is "sitting", the minimum edit distance (i.e. the minimum edit times) between "kitten" and "sitting" can be calculated by the edit distance formula as 3, including replacing k with s, inserting i, and replacing e with i, and further, the normalized text similarity is 57.14%, so that by calculating the edit distance between different text information and converting it into the corresponding text similarity, the text similarity between the text displayed on the mobile terminal and the central control screen can be accurately and effectively calculated.

[0099] For the icon similarity, the first icon information includes a first icon region and the second icon information includes a second icon region, the icon region can be a region intercepted in the control interface, and the icon region includes corresponding icons, such as icons corresponding to interactive controls, etc., so that the first pixel value of each first pixel point in the first icon region can be obtained, the second pixel value of each second pixel point in the second icon region can be obtained, and the stability constant for the first icon region and the second icon region can be obtained, and then the icon similarity between the first image and the second image can be obtained by calculating according to the first pixel value, the second pixel value and the stability constant, so that by calculating the similarity between different icon information displayed on the mobile terminal and the central control screen, the difference between the two can be analyzed from the icon dimension. The stability constant can be a constant for setting the stability in the calculation process, which is not limited by the present application.

[0100] In a specific implementation, the first average value corresponding to the first pixel value and the first variance can be calculated, and the second average value corresponding to the second pixel value and the second variance can be calculated, then the covariance between the first variance and the second variance can be calculated, and finally the first average value, the second average value, the first variance, the second variance, the covariance and the stability constant can be used to calculate the icon similarity between the first image and the second image.

[0101] The average value based on the pixel value can effectively reflect the brightness of the image, and the variance based on the pixel value can effectively reflect the contrast of the image, and the covariance can reflect the structural correlation between the two images, so that by calculating the relevant parameters and performing comprehensive calculation, the brightness, contrast and structural information of the image can be comprehensively compared to determine the icon similarity between the two images, and the comparison in the icon dimension is accurately and effectively realized.

[0102] In some examples, the icon similarity can be calculated by the following formula:

[0103]

[0104] Wherein, SSIM(x, y) represents a structural similarity index, i.e. the structural similarity between the two icon regions, and the value range can be -1 to 1; μ x represents the average value of the pixels of the first icon region x, and the average value can reflect the brightness of the first icon region; y represents the average value of the pixels of the second icon region y, and the average value can reflect the brightness of the second icon region; represents the pixel variance of the first icon region x, and the pixel variance can reflect the contrast of the first icon region; represents the pixel variance of the second icon region y, and the pixel variance can reflect the contrast of the second icon region; σ xy is the covariance of the pixel values between the first icon region x and the second icon region y, and the covariance can reflect the structural correlation between the two images); c1 can be a first stable constant, which can be a small positive number, usually (0.01*L)^2, L is the dynamic range, to prevent the denominator from being 0; c2 can be a stable constant 2, which can be a small positive number, usually (0.03*L)^2, to further stabilize the calculation; 2 μ x μ y +c1 can be used to represent the brightness similarity term, i.e. the weighted sum of enhancing the brightness contrast; 2 σ xy +c1 represents the structural similarity term, i.e. the weighted sum of enhancing the structural correlation, and through the formula, the brightness, contrast and structural information of the image can be comprehensively compared to determine the icon similarity between the two icon regions, which accurately and effectively realizes the comparison of the icon dimension, and further determines the similarity of the two images in the icon dimension.

[0105] After obtaining the text similarity and the icon similarity through the above process, a first weight value corresponding to the text similarity can be further obtained, and a second weight value corresponding to the icon similarity can be further obtained, and then the text similarity, the first weight value, the icon similarity and the second weight value are calculated to obtain the interface difference degree between the first image and the second image, so that on the one hand, the consistency test can be carried out based on a single dimension, and on the other hand, the interface consistency test can be carried out from multiple dimensions, so that the differences between different interfaces can be comprehensively and accurately captured, and the accuracy of the test is ensured.

[0106] For example, the interface difference degree can be calculated by the following formula:

[0107] Total Similarity=w1×Text Similarity+w2×Icon Similarity

[0108] The Total Similarity represents interface difference, the Text Similarity represents text similarity, the Icon Similarity represents icon similarity, w1 represents a first weight value, and w2 represents a second weight value. The text similarity and the icon similarity are comprehensively processed, the interface consistency is tested from multiple dimensions, the differences between different interfaces are comprehensively and accurately captured, and the accuracy of the test is ensured.

[0109] In step 103, an interface detection report for the central control screen is output according to the interface difference between the mobile terminal and each central control screen.

[0110] After obtaining the interface difference between the mobile terminal and each central control screen, the corresponding interface detection report can be output based on the interface difference, so that when the interface consistency of different devices is detected, for the interface display content at the same time, the corresponding image information is extracted, and the interface difference is determined based on the image information, the process of the interface consistency test is simplified, the test efficiency is improved, and by matching the content displayed by at least one mobile terminal with the content displayed by at least two central control screens, batch testing can be realized, and the test efficiency is further improved.

[0111] In some possible implementation manners, the average difference between the interface differences can be calculated first, then the standard deviation between the interface differences and the average difference is calculated, the average difference and the standard deviation are used for calculation to obtain a target detection value, then the interface differences and the target detection value can be compared to determine the difference between the content displayed by the central control screen and the content limited by the mobile terminal, the central control screen with an interface difference greater than or equal to the target detection value is regarded as an abnormal device, and an abnormal device mark corresponding to the abnormal device is generated. Through secondary comparison, the accuracy of the interface test is further improved. In addition, the interface detection report for the central control screen can be output based on one of the interface difference and the abnormal device mark, so that the tester can intuitively view the interface consistency between different central control screens and the mobile terminal, thereby facilitating the subsequent development of the central control screen.

[0112] It should be noted that the target detection value can be used as a detection index for overall consistency, and batch and secondary detection can be realized through the detection index, thereby improving the accuracy of the interface consistency test.

[0113] In some examples, the average difference corresponding to each central control screen can be calculated by the following formula:

[0114]

[0115] Wherein, the Mean Difference represents an average difference degree, which can be used to evaluate multi-device consistency; N can be a sample number, i.e., a total number of the central control screen or the device to be tested; d i Can be the i-th interface difference degree, i.e., the interface difference degree between the i-th central control screen and the mobile terminal; Σd i Represents a sum of all difference degrees; 1 / N represents an average factor, and the corresponding average difference degree can be calculated through the formula, so that the average difference degree can be used as an overall consistency index.

[0116] Further, the corresponding standard deviation can be calculated through the following formula:

[0117]

[0118] Through the formula, the dispersion degree of the interface difference degree corresponding to each central control screen can be analyzed, so as to identify the interface abnormal abnormal central control screen from all central control screens based on the standard deviation.

[0119] Optionally, after obtaining the standard deviation, the sum of the average difference degree and 1.5 times the standard deviation can be calculated, and then the sum value is used as a target detection value, and the interface difference degree corresponding to each central control screen is compared with the target detection value, and the central control screen exceeding the target detection value is screened out, and these central control screens are used as abnormal devices, so that the test personnel can process the abnormal devices.

[0120] It should be noted that the embodiments of the present application include but are not limited to the above examples, and it can be understood that those skilled in the art can also set according to actual needs under the guidance of the idea of the embodiments of the present application, and the present application does not limit this.

[0121] In the embodiment of the application, in the consistency detection process of the interface display between the central control screen and the mobile terminal, the central control screen and the mobile terminal can be accessed into the same smart home device to ensure the consistency of device control. The first image corresponding to the first graphical user interface and the second image corresponding to the second graphical user interface are obtained, the first image and the second image are images obtained by triggering the screenshot at the same time, then the first image information is extracted from the first image and the second image information is extracted from the second image, and the similarity of the first image information and the second image information is matched to obtain the interface difference degree between the first image and the second image. Then, according to the interface difference degree between the mobile terminal and each central control screen, an interface detection report for the central control screen is output, so that when the interface consistency of different devices is detected, the interface display content at the same time is extracted for the corresponding image information, and the corresponding interface difference degree is determined based on the image information, which simplifies the process of interface consistency test, improves the test efficiency, and through matching the content displayed by at least one mobile terminal with the content displayed by at least two central control screens, batch testing can be realized, further improving the test efficiency.

[0122] In order for those skilled in the art to better understand the technical solutions in the embodiments of the application, the following will be exemplarily described by corresponding examples:

[0123] As an example, a corresponding test system can be constructed, which can at least include

[0124] ① Central control screen module: multiple Linux system central control screens (for example, Screen1, Screen2, …, ScreenN), which transmit screenshots to the PC end through a local area network.

[0125] ② Mobile terminal module: mobile phone application program, which is used as a reference for detecting whether the interface display of the central control screen is normal.

[0126] ③ PC end processing module:

[0127] By receiving the access sent by the central control screen module and the mobile terminal module, all screenshots are stored, and then the difference degree of each central control screen and the App is calculated, and the consistency of multiple devices is analyzed to generate a report.

[0128] Optionally, the corresponding test method steps can include:

[0129] 1. Initialization:

[0130] Access the central control screen and the mobile terminal into the same smart home system to ensure that the device state is synchronized.

[0131] Create a storage directory on the PC end, and name the screenshot file according to the central control screen ID and the timestamp.

[0132] 2. Screenshot Collection:

[0133] Control Screen Screenshot:

[0134] Run a preset script (e.g., using the `scrot` tool) on each control screen to periodically or on-demand capture the current interface image.

[0135] Transfer the screenshot to the designated directory on the PC via the SCP protocol.

[0136] Mobile Terminal Screenshot:

[0137] Use Appium to connect to the mobile terminal, call the screenshot interface (e.g., `get_screenshot_as_file`), and save it to the same directory on the PC.

[0138] 3. Data Extraction:

[0139] Extract the display content of the corresponding area from the screenshots of the control screens and mobile terminals:

[0140] Text Extraction: Use an OCR (Optical Character Recognition) tool (such as Tesseract) to recognize the text in the specified area (such as the device name in the top-left corner).

[0141] Icon State Extraction: Analyze the color and brightness of the icon area using OpenCV to determine the highlight or grayscale state.

[0142] 4. Difference Calculation:

[0143] A) Text Difference: Calculate the text matching rate between each control screen and the mobile terminal application.

[0144] Calculate the text similarity based on the Levenshtein distance (i.e., edit distance).

[0145] 1) Extract text using Tesseract OCR (supports Chinese, `lang='chi'`).

[0146] 2) Calculate the Levenshtein distance between the control screen text and the mobile terminal application text:

[0147]

[0148] Convert to matching rate:

[0149]

[0150] The result ranges from 0 to 1, multiplied by 100 to get the percentage.

[0151] B) Icon Difference: Calculate the icon similarity between each of the control screens and the mobile app. Calculate the icon similarity based on Structural Similarity Index (SSIM).

[0152] 1) Crop the icon region (e.g. [150:200, 300:350], etc.).

[0153] 2) Calculate SSIM (Structural Similarity Index Measure):

[0154]

[0155] C) Overall Difference: Combine the text and icon states to generate an overall difference score for each of the control screens.

[0156] Use weighted average method.

[0157] 1) Set the weights for text and icon states (e.g. 0.6 and 0.4).

[0158] 2) Calculate the overall difference (i.e. interface difference):

[0159] Total Similarity = w1 x Text Similarity + w2 x Icon Similarity

[0160] 5. Multi-device Consistency Analysis:

[0161] Calculate the average difference between all control screens and the mobile app.

[0162] Identify devices with abnormal difference (e.g. beyond the range of standard deviation).

[0163] 1) Calculate the list of differences for all control screens (e.g. [d1, d2,..., dN]).

[0164] 2) Calculate the average difference:

[0165]

[0166] 3). Calculate the standard deviation:

[0167]

[0168] 4) Mark abnormal devices: Devices with difference beyond the mean ± 1.5 times the standard deviation.

[0169] 6. Results Output:

[0170] generating a report including a difference score for each central control screen, an overall consistency analysis, and an abnormal device marker.

[0171] Through the above process, when detecting the interface consistency of different devices, for the interface display content at the same time, the corresponding image information is extracted, and the interface difference degree is determined based on the image information, which simplifies the process of interface consistency test, improves the test efficiency, and through matching the content displayed by at least one mobile terminal with the content displayed by at least two central control screens, batch testing can be realized, further improving the test efficiency.

[0172] It should be noted that for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited by the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0173] Referring to Figure 3 , a structure block diagram of a processing device of an interface provided in an embodiment of the present application is shown, which relates to at least one mobile terminal and at least two central control screens, the mobile terminal and the central control screens are connected to the same smart home device; the mobile terminal provides a first graphical user interface, and the central control screen provides a second graphical user interface, which can specifically include the following modules:

[0174] The image acquisition module 301 is configured to acquire a first image corresponding to the first graphical user interface and a second image corresponding to the second graphical user interface, the first image and the second image are images obtained by triggering screen capture at the same time;

[0175] The test module 302 is configured to extract first image information from the first image and second image information from the second image, and perform similarity matching according to the first image information and the second image information to obtain the interface difference degree between the first image and the second image;

[0176] The output module 303 is configured to output an interface detection report for the central control screen according to the interface difference degree between the mobile terminal and each central control screen.

[0177] In some possible implementation manners, the first image information at least includes first text information and first icon information, the second image information at least includes second text information and second icon information, and the test module 302 is specifically configured to:

[0178] matching the first text information with the second text information to determine a text similarity;

[0179] matching the first icon information with the second icon information to determine an icon similarity;

[0180] calculating based on the text similarity and the icon similarity to obtain an interface difference degree between the first image and the second image.

[0181] In some possible implementation manners, the test module 302 is specifically configured to:

[0182] matching a first character in the first text information and the second text information to calculate an edit distance of the second text information converted into the first text information in a matching process;

[0183] obtaining a first text length of the first text information and a second text length of the second text information, and converting the first text length, the second text length and the edit distance to obtain a text similarity between the first image and the second image.

[0184] In some possible implementation manners, the first icon information includes a first icon region, and the second icon information includes a second icon region, and the test module 302 is specifically configured to:

[0185] obtaining a first pixel value of each first pixel point in the first icon region;

[0186] obtaining a second pixel value of each second pixel point in the second icon region;

[0187] obtaining a stable constant for the first icon region and the second icon region;

[0188] calculating according to the first pixel value, the second pixel value and the stable constant to obtain an icon similarity between the first image and the second image.

[0189] In some possible implementation manners, the test module 302 is specifically configured to:

[0190] calculating a first average value and a first variance corresponding to the first pixel value;

[0191] calculating a second average value and a second variance corresponding to the second pixel value;

[0192] calculating a covariance between the first variance and the second variance;

[0193] The first average value, the second average value, the first variance, the second variance, the covariance and the stability constant are used for calculation to obtain an icon similarity between the first image and the second image.

[0194] In some possible implementation manners, the test module 302 is specifically configured to:

[0195] obtain a first weight value corresponding to the text similarity;

[0196] obtain a second weight value corresponding to the icon similarity;

[0197] The text similarity, the first weight value, the icon similarity and the second weight value are used for calculation to obtain an interface difference degree between the first image and the second image.

[0198] In some possible implementation manners, the output module 303 is specifically configured to:

[0199] calculate an average difference degree between each interface difference degree;

[0200] calculate a standard deviation between each interface difference degree and the average difference degree;

[0201] The average difference degree and the standard deviation are used for calculation to obtain a target detection value;

[0202] a central control screen with an interface difference degree greater than or equal to the target detection value is regarded as an abnormal device, and an abnormal device mark corresponding to the abnormal device is generated;

[0203] An interface detection report for the central control screen is output based on one of the interface difference degree and the abnormal device mark.

[0204] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0205] In addition, the embodiment of the present application further provides an electronic device, which comprises a processor, a memory, a computer program stored in the memory and executable on the processor, the computer program implements each process of the interface processing method embodiment of the present application when executed by the processor, and the same technical effects can be achieved, to avoid repetition, which will not be described here.

[0206] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to realize each process of the processing method embodiment of the interface, and the same technical effects can be achieved. To avoid repetition, details are not described herein. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0207] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment are referred to each other.

[0208] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device or computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash and eMMC, etc.) containing computer usable program codes.

[0209] The embodiments of the present application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device for realizing the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for realizing the functions specified in one or more flows and / or blocks. Figure 1 The device for realizing the functions specified in one or more flows and / or blocks.

[0210] These computer program instructions can also be stored in a computer readable storage medium to guide the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which realize the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for realizing the functions specified in one or more flows and / or blocks. Figure 1 The device for realizing the functions specified in one or more flows and / or blocks.

[0211] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the flowchart Figure 1 one block or multiple blocks.

[0212] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to cover all changes and modifications falling within the scope of the embodiments of the present application.

[0213] Finally, it should also be noted that, in this document, the relational terms such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between or among the entities or actions. Also, the terms "comprises", "comprising", or any other variations thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or terminal device including a list of elements does not include only those elements but can include other elements not expressly listed or even include elements inherent in such process, method, article, or terminal device. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or terminal device including the element.

[0214] The above describes in detail the processing method of an interface and the processing device of an interface provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A method of processing an interface, characterized by, The method relates to at least one mobile terminal and at least two central control screens, the mobile terminal and the central control screens being connected to the same smart home device; The mobile terminal provides a first graphical user interface, and the central control screen provides a second graphical user interface, and the method comprises the following steps: obtaining a first image corresponding to the first graphical user interface and a second image corresponding to the second graphical user interface, the first image and the second image being images obtained by triggering screenshot at the same time; extracting first image information from the first image and second image information from the second image, and performing similarity matching on the first image information and the second image information to obtain the interface difference degree between the first image and the second image; outputting an interface detection report for the central control screen according to the interface difference degree between the mobile terminal and each central control screen.

2. The method of claim 1, wherein, The first image information at least comprises first text information and first icon information, the second image information at least comprises second text information and second icon information, and the similarity matching on the first image information and the second image information to obtain the interface difference degree between the first image and the second image comprises the following steps: matching the first text information with the second text information to determine the text similarity; matching the first icon information with the second icon information to determine the icon similarity; calculating the text similarity and the icon similarity to obtain the interface difference degree between the first image and the second image.

3. The method of claim 2, wherein, The matching of the first text information with the second text information to determine the text similarity comprises the following steps: matching the first character in the first text information with the first character in the second text information to calculate the edit distance of the second text information converted into the first text information in the matching process; obtaining the first text length of the first text information and the second text length of the second text information, and converting the first text length, the second text length and the edit distance to obtain the text similarity between the first image and the second image.

4. The method according to claim 2 or 3, characterized in that, The first icon information comprises a first icon region, and the second icon information comprises a second icon region, and the matching of the first icon information with the second icon information to determine the icon similarity comprises the following steps: obtaining the first pixel value of each first pixel point in the first icon region; obtaining the second pixel value of each second pixel point in the second icon region; obtaining a stable constant for the first icon region and the second icon region; calculating the first pixel value, the second pixel value and the stable constant to obtain the icon similarity between the first image and the second image.

5. The method of claim 4, wherein, The calculation of the first pixel value, the second pixel value and the stable constant to obtain the icon similarity between the first image and the second image comprises the following steps: calculating the first average value and the first variance corresponding to the first pixel value; calculating the second average value and the second variance corresponding to the second pixel value; calculating a covariance between the first variance and the second variance; calculating, using the first average, the second average, the first variance, the second variance, the covariance, and the stability constant, an icon similarity between the first image and the second image.

6. The method of claim 2, wherein, calculating, based on the text similarity and the icon similarity, an interface difference between the first image and the second image, includes: obtaining a first weight value corresponding to the text similarity; obtaining a second weight value corresponding to the icon similarity; calculating, using the text similarity, the first weight value, the icon similarity, and the second weight value, an interface difference between the first image and the second image.

7. The method according to claim 1 or 6, characterized in that, outputting, according to the interface difference between the mobile terminal and each of the central control screens, an interface detection report for the central control screens, includes: calculating an average difference between each of the interface differences; calculating a standard deviation between each of the interface differences and the average difference; calculating, using the average difference and the standard deviation, a target detection value; regarding a central control screen with an interface difference greater than or equal to the target detection value as an abnormal device, and generating an abnormal device label corresponding to the abnormal device; outputting, based on one of the interface difference and the abnormal device label, an interface detection report for the central control screen.

8. A processing device of an interface, characterized in that relating to at least one mobile terminal and at least two central control screens, the mobile terminal and the central control screens being connected to the same smart home device; the mobile terminal provides a first graphical user interface, and the central control screen provides a second graphical user interface, and the device includes: an image acquisition module, configured to acquire a first image corresponding to the first graphical user interface and a second image corresponding to the second graphical user interface, the first image and the second image being images obtained by triggering screen capture at the same time; a test module, configured to extract first image information from the first image, extract second image information from the second image, and perform similarity matching according to the first image information and the second image information to obtain an interface difference between the first image and the second image; an output module, configured to output, according to the interface difference between the mobile terminal and each of the central control screens, an interface detection report for the central control screen.

9. An electronic device, comprising: comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; the memory is used to store a computer program; the processor is used to execute the program stored on the memory, and implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, instructions stored thereon, when executed by one or more processors, cause the processors to perform the method of any one of claims 1-7.

Images