Visual restoration detection method and device

Abstract

The invention provides a visual restoration detection method and device. The visual restoration detection method and device are applied to a visual detection client corresponding to a developed target application. The method comprises the steps of obtaining a visual design file corresponding to a target application, and analyzing the visual design file to obtain a visual effect picture corresponding to an application interface of the target application; obtaining an interface screenshot of the application interface output to a user by the target application in a running process; and calling a detection algorithm on the visual detection client, calculating different page elements in the interface screenshot and the visual effect picture, and labeling the different page elements on the interface screenshot.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology. More specifically, the embodiments of the present specification relate to a visual restoration detection method and device. Background Art

[0002] The user interface is a key part of the software design process. It is the medium for interaction and information exchange between the software system and the user. It realizes the conversion between the internal form of information and the form acceptable to humans. The process of development engineers restoring the visual design draft of the user interface provided by the designer during the development phase, that is, the accuracy of visual restoration, directly affects the user experience and needs to be carefully checked.

[0003] For user interfaces displayed in graphical form, visual restoration detection is a method of checking the visual restoration effect of the user interface. It can be used to determine whether the designed interface can interact normally and whether the data in the user interface is correctly displayed and designed.

[0004] Existing visual restoration inspection of graphical user interfaces is often carried out manually, relying on the experience of inspectors and test cases. This inspection method is time-consuming and labor-intensive, inefficient, and has a poor error detection rate. Summary of the invention

[0005] To overcome the problems existing in the related art, this specification provides the following methods and devices.

[0006] In a first aspect of the implementation manner of this specification, a visual restoration detection method is provided, which is applied to a visual detection client corresponding to a developed target application; the method comprises:

[0007] Acquire a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to an application interface of the target application;

[0008] Obtaining a screenshot of the application interface output to the user during the operation of the target application;

[0009] The detection algorithm on the visual detection client is called to calculate the page elements that are different between the interface screenshot and the visual effect image, and the page elements that are different are marked on the interface screenshot.

[0010] In a second aspect of the embodiments of the present specification, a visual restoration detection device is provided, which is applied to a visual detection client corresponding to a developed target application; the device comprises:

[0011] A parsing unit, used to obtain a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to an application interface of the target application;

[0012] A screenshot unit, used to obtain a screenshot of the application interface output to the user during the operation of the target application;

[0013] A calculation unit is used to call the detection algorithm on the visual detection client, calculate the page elements that are different between the interface screenshot and the visual effect image, and mark the page elements that are different on the interface screenshot.

[0014] In a third aspect of the embodiments of the present specification, a visual restoration detection system is provided, the system comprising:

[0015] Vision inspection client for:

[0016] Acquire a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to an application interface of the target application;

[0017] Receive a screenshot of the target application interface sent by the test client, call a detection algorithm, calculate the page elements that are different between the screenshot and the visual effect image, and mark the page elements that are different on the screenshot;

[0018] Test client for:

[0019] The target application is run, and a screenshot of the target application interface output to the user during the running of the target application is obtained and sent to the visual inspection client.

[0020] In a fourth aspect of the embodiments of the present specification, a storage medium is provided; a computer program is stored on the storage medium, and the computer program implements the steps of the method described above when executed.

[0021] In a fifth aspect of the embodiments of the present specification, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above method is implemented when the processor executes the program.

[0022] The above embodiments of this specification have at least the following beneficial effects:

[0023] In the above technical scheme, a detection algorithm is used to calculate the visual effect diagram of the visual design file of the running interface of the target application and the interface screenshot when the target application is actually running, and the page elements of the above interface screenshots and the visual effect diagram are compared. The page elements with differences are marked on the interface screenshots to realize real-time detection of the visual restoration of the running interface of the target application, so that developers can intuitively understand the implementation effect of visual restoration and reduce the collaboration cost with designers. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification, and those of ordinary skill in the art can also obtain other drawings based on these drawings.

[0025] Figure 1 The schematic diagram schematically shows a structure of a visual restoration detection method according to an embodiment of the present specification;

[0026] Figure 2 A flowchart of a visual restoration detection method according to an embodiment of the present specification is schematically shown;

[0027] Figure 3 A schematic diagram schematically shows a visual restoration detection method according to an embodiment of the present specification;

[0028] Figure 4 A schematic diagram schematically shows a visual restoration detection method according to an embodiment of the present specification;

[0029] Figure 5 A schematic diagram of a connection of a visual restoration detection method according to an embodiment of the present specification is schematically shown;

[0030] Figure 6 A schematic diagram of a visual restoration detection method according to an embodiment of the present specification is schematically shown;

[0031] Figure 7 A schematic diagram of a report of a visual restoration detection method according to an embodiment of the present specification is schematically shown;

[0032] Figure 8 A block diagram of a visual restoration detection device according to an embodiment of the present specification is schematically shown;

[0033] Fig. 9 The structure diagram of a visual restoration detection system according to an embodiment of the present specification is schematically shown;

[0034] Fig.10 A hardware structure diagram of a computer device in which a visual restoration detection method according to an embodiment of the present specification is located is schematically shown.

[0035] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0036] The principles and spirit of this specification will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement this specification, and are not intended to limit the scope of this specification in any way. On the contrary, these embodiments are provided to make this specification more thorough and complete, and to fully convey the scope of this specification to those skilled in the art.

[0037] Those skilled in the art know that the embodiments of this specification can be implemented as a system, device, equipment, method or computer program product. Therefore, this specification can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0038] First, let’s introduce some related concepts that may be involved in the manual:

[0039] Visual restoration: refers to the process in which development engineers restore the elements, fonts, spacing and other styles of the visual design drafts provided by designers during the development phase of Internet software applications.

[0040] Visual walkthrough: refers to the process in which product managers, designers, etc. conduct an interactive visual review of software functions when an Internet software application is submitted for acceptance after development.

[0041] Electron: Electron is an open source development framework that uses HTML, CSS, and JavaScript to build cross-platform desktop applications. It can complete the development of cross-platform desktop GUI applications by using Node.js (as the backend) and Chromium's rendering engine (as the frontend).

[0042] SketchTool: SketchTool is a command-line utility bundled with Sketch, which allows performing some operations with Sketch documents, such as inspecting them or exporting assets, etc.

[0043] WebSocket technology: The WebSocket protocol is a new network protocol based on TCP. It implements full-duplex communication between the browser and the server, that is, it allows the server to actively send information to the client. Therefore, after the connection channel is established, data can be transmitted in both directions.

[0044] OpenCV: OpenCV is an open source computer vision library. It consists of a series of C functions and a small number of C++ classes, which implement many common algorithms in image processing and computer vision.

[0045] As the quality requirements for interactive visuals in terminal applications have gradually increased in recent years, visual restoration work has also occupied a large proportion in the application development process. On the one hand, terminal development at this stage lacks tool support to efficiently restore page effects. On the other hand, when designers conduct visual walkthroughs, frequent problems lead to low efficiency in collaboration between developers and designers. How to improve the accuracy of visual restoration and reduce collaboration costs is a pain point in this process, and it is also a common problem faced by terminal application developers in the industry.

[0046] In the related technology, the renderings can be obtained by parsing the web design sketch file on the server, and then compared with the web screenshots uploaded by the developer, and the image can be compared through the comparison program on the server to find the differences between the web screenshots and the renderings, thereby completing the visual restoration test.

[0047] However, this method is based on a comparison program on the server, which places a large strain on server resources and has high operating costs. It also requires developers to take screenshots in advance and upload them, which is inconvenient to use.

[0048] To overcome the problems existing in the related art, this specification provides the following embodiments.

[0049] The scheme of this specification is described in detail below with reference to the accompanying drawings.

[0050] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the architecture of a visual restoration detection system provided by an exemplary embodiment. Figure 1 As shown, the system may include a network 10, a test device 11, a detection device 12, and the like.

[0051] Network 10 may include various types of wired or wireless networks.

[0052] The test device 11 can be a dedicated test device, or a general device that meets relevant functions, such as a mobile phone, a computer, an embedded device, etc. The test device 11 can run the application to be tested and output the application interface to the user.

[0053] The detection device 12 may be a device associated with the test device 11. Specifically, the detection device 12 may establish a connection with the test device 11 via the network 10. The detection device 12 may be a computer supporting an operating system such as Windows / Mac / Linux, and one or at least one embodiment of this specification is not limited thereto.

[0054] In one embodiment, the detection device 12 can cooperate with the test device 11; wherein the detection device 12 can accept user operations and upload commands to the test device 11 through the network 10, and then the test device 11 cooperates with the detection device 12 to process files based on the solution of this specification. In another embodiment, the detection device 12 can independently implement the solution of this specification; wherein the detection device 12 accepts user operations and processes the accepted commands and files based on the solution of this specification to achieve visual restoration detection.

[0055] See also Figure 2 , Figure 2 is a flowchart of a visual restoration detection method provided by an exemplary embodiment, the method is applied to a processing device, and the processing device can be Figure 1 The detection device 12 shown, etc.

[0056] The method is applied to the visual inspection client corresponding to the developed target application;

[0057] The method may include the following steps:

[0058] Step 202: Obtain a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to the application interface of the target application.

[0059] In software design, visual restoration usually refers to restoring the visual design draft created by the designer (UI / UX designer) into the actual software interface or user interface (UI for short). This is a process of transforming a static design into a dynamic, interactive practical application.

[0060] When software designers complete the visual design drafts of an application or website, these design drafts are usually provided in the form of visual design files (such as PSD, Sketch, Figma, etc.). Then, developers will perform a visual restoration process based on these visual design drafts and convert them into actual working user interfaces.

[0061] In the process of visual restoration, the effect of visual restoration needs to be tested. However, it is often difficult to find some subtle differences by the naked eye of developers alone. In addition, the terminal where the developed target application is actually run or tested is often not the terminal where the developer develops the application. For example, the development terminal is a PC, and the terminal where the target application is run and tested is a mobile phone. Therefore, it is not convenient to make a direct comparison. Therefore, it is necessary to compare the result of visual restoration (the interface when the target application is running) and the effect that the visual restoration wants to achieve (the design draft provided by the designer) on the same terminal.

[0062] First, you can get the visual design draft provided by the designer, usually in the form of a visual design file (such as PSD, Sketch, Figma, etc.).

[0063] A visual file parsing component can be installed on the visual inspection client to parse visual design files.

[0064] By parsing the acquired design visual file, a visual effect diagram corresponding to the application interface of the target application can be obtained.

[0065] For example, when the visual design file is in Sketch format, the SketchTool module can be used to parse it to obtain the corresponding visual effect diagram.

[0066] When the visual design file includes visual design drafts of multiple application interfaces corresponding to the target application, the visual file parsing component can parse out visual design renderings corresponding to the multiple application interfaces.

[0067] In this specification, the target application can be a client application running on various electronic devices such as PCs, mobile phones, embedded devices, etc., or a web application, and this specification does not specifically limit this.

[0068] Step 204: Obtain a screenshot of the application interface output to the user during the operation of the target application.

[0069] To perform visual restoration detection, in addition to the visual effect graph, it is also necessary to obtain the application interface output to the user during the actual operation of the target application obtained by visual restoration. Usually, the interface screenshot can be obtained by taking a screenshot of the application interface corresponding to the visual effect graph of the target application.

[0070] After obtaining the interface screenshot, you can compare it with the corresponding visual effect diagram to perform visual restoration detection.

[0071] If the target application is running on the visual inspection client, the application interface output to the user can be directly screenshotted. If the target application is running on a test terminal other than the visual inspection client, the screenshot can be completed on the test terminal and sent to the visual inspection client.

[0072] Step 206: calling the detection algorithm on the visual detection client, calculating the page elements that are different between the interface screenshot and the visual effect image, and marking the page elements that are different on the interface screenshot.

[0073] After obtaining the visual effect image of the target application and the interface screenshot of the application interface output to the user corresponding to the visual effect image, they can be compared.

[0074] A detection algorithm may be preset on the visual detection client to calculate the above-mentioned visual effect image and its corresponding interface screenshot to obtain page elements that are different between the interface screenshot and the visual effect image.

[0075] A page element can be any component on the application interface of the target application, such as a picture, text, or form.

[0076] The preset detection algorithm may be a traditional pixel-based comparison algorithm or an artificial intelligence-based image recognition and image comparison algorithm, and this specification does not specifically limit this.

[0077] After the detection algorithm is used to calculate the page elements that are different between the interface screenshot and the visual effect image, the different page elements can be marked on the interface screenshot, for example, by highlighting, adding a frame, etc., which is not specifically described in this description.

[0078] See also Figure 3 , Figure 3 It is a schematic diagram of a visual restoration detection method according to an implementation method of this specification.

[0079] In the visual inspection method shown in this specification, the visual effect diagram is parsed through the original visual design file provided by the designer;

[0080] Correspondingly, an interface screenshot is obtained by taking a screenshot of the running interface output by the target application to the user.

[0081] After obtaining the interface screenshot corresponding to the visual effect image, call the image algorithm comparison module to compare the interface elements in the interface screenshot and the visual effect image, and mark the interface elements with differences in the interface screenshot.

[0082] The above embodiment uses a detection algorithm to calculate the visual effect diagram of the visual design file of the running interface of the target application and the interface screenshot when the target application is actually running, compares the page elements of the above interface screenshots and the visual effect diagram, and marks the page elements with differences on the interface screenshots to achieve real-time detection of the visual restoration of the running interface of the target application, so that developers can intuitively understand the implementation effect of the visual restoration and reduce the collaboration cost with designers.

[0083] Moreover, in this solution, the visual inspection client can run on the developer's local device, does not rely on centralized server computing resources, and is more flexible.

[0084] In addition to using detection algorithms to calculate the page elements that are different, a more intuitive way can be used to show developers the differences between interface screenshots and visual renderings.

[0085] After obtaining the interface screenshot and the visual effect image, they can be adjusted to the same size, and the interface screenshot can be made transparent and superimposed with the corresponding visual effect image. The superimposed image can be displayed to the developer for visual comparison.

[0086] There is no doubt that if there is no difference between the interface screenshot and the visual effect diagram, after the transparent interface screenshot is superimposed on the corresponding visual effect diagram, the page elements on the visual effect diagram will not change;

[0087] Different page elements on the interface screenshot and the visual effect diagram will be misaligned after the above-mentioned superposition, resulting in obvious differences in the superimposed images. Developers can usually observe these differences intuitively.

[0088] In addition, the interface screenshots can be inverted and overlaid with the visual effect images.

[0089] Undoubtedly, the parts that are the same in the interface screenshot and the visual effect picture will be black after inversion and superposition, while the parts that are different in the interface screenshot and the visual effect picture may appear in other colors.

[0090] The target application may be run on the above-mentioned visual inspection client, or may be run on a test client different from the above-mentioned visual inspection client.

[0091] When the target application runs on a test client different from the visual inspection client, a connection can be established between the visual inspection client and the test client on which the target application runs, so that the visual test client can obtain the above interface screenshots from the test client.

[0092] In order to obtain the above-mentioned interface screenshots conveniently, stably and in real time, a long connection can be established between the visual inspection client and the test client running the target application.

[0093] Long connection can keep the connection status instead of closing it immediately after one communication, so that the visual inspection client and the test client can keep the communication connection and exchange data multiple times until the connection is closed after a certain condition is met. This method can reduce the overhead caused by frequent establishment and closing of connections and improve the efficiency and performance of communication.

[0094] Specifically, a long connection between the visual inspection client and the test client can be established through TCP long connection, long polling connection, WebSocket connection, MQTT connection, etc., which is not specifically limited in this description.

[0095] After a persistent connection is established between the visual inspection client and the test client, the visual inspection client can obtain from the test client a screenshot of the application interface output by the target application to the user during operation, so as to perform visual inspection on the visual inspection client.

[0096] In an exemplary embodiment shown in this specification, the detection algorithm on the visual detection client may include an image recognition algorithm and an image comparison algorithm.

[0097] Wherein, the image recognition algorithm is used to identify the page elements in the interface screenshot and the visual effect diagram;

[0098] Commonly used image recognition algorithms may include: image recognition algorithms based on convolutional neural networks (CNN), image recognition algorithms based on support vector machines (SVM), etc.

[0099] Usually, Python-based image recognition algorithms can be used. Python has many powerful libraries and frameworks in the field of image recognition, which can implement various image recognition algorithms, such as OpenCV, TensorFlow, Keras, etc.

[0100] The image comparison algorithm is used to calculate the page elements that are different between the interface screenshot and the visual effect image.

[0101] Generally, an image comparison algorithm based on an OpenCV framework, an image comparison algorithm based on a mean square error (MSE), an image comparison algorithm based on a hash algorithm, or the like may be used as the image comparison algorithm.

[0102] Using image recognition algorithms for calculations can improve the accuracy of visual restoration and reduce collaboration costs.

[0103] After the comparison and annotation are completed, a visual inspection report, or visual walkthrough report, can be generated for the developer to summarize the conclusions of the visual inspection. The developer can export the report to understand the visual restoration situation, or send it to the designer for visual restoration approval.

[0104] The visual walkthrough report should at least include screenshots of the interface with the page elements that are different marked.

[0105] In addition to the above-mentioned interface screenshots, it may also include information in multiple dimensions, such as the number of different page elements in each page, the original screenshot file, the visual effect diagram, and the similarity score of the interface screenshot relative to the visual effect diagram.

[0106] The similarity score can be calculated according to a preset similarity score algorithm, and the specific similarity score algorithm is not specifically limited in this specification. For example, the similarity of the images can be quantified using an index such as mean square error or structural similarity index, and normalized into a similarity score in percentage.

[0107] See also Figure 4 , Figure 4 It is a schematic diagram of a visual restoration detection method according to an implementation method of this specification.

[0108] Figure 4 The specific process of a visual restoration detection method is demonstrated.

[0109] First, download and install the above visual inspection client in the developer's visual restoration inspection environment. The client can support different operating systems such as Mac / Windows.

[0110] The visual inspection client can be developed based on a cross-platform framework and supports real-time connection to mobile apps or other device terminals.

[0111] During implementation, a visual restoration detection project corresponding to the target application may be established first.

[0112] After creating a project, you can import visual design files provided by designers, such as visual design files in Sketch format.

[0113] The visual inspection client can call SketchTool to locally parse the visual design file in the above Sketch format to obtain a visual effect diagram.

[0114] See also Figure 5 , Figure 5 A connection diagram of a visual restoration detection method according to an embodiment of the present specification is schematically shown.

[0115] When a Sketch file corresponds to the visual design drafts of multiple application interfaces, multiple corresponding visual effect diagrams can be parsed.

[0116] like Figure 5 As shown, the visual inspection client analyzes the visual design draft to parse out the visual effect images corresponding to multiple application interfaces such as "Minimal Conversation", "Music Label", and "Radio" in the target application.

[0117] The target application runs on the test client, which can be on a mobile phone.

[0118] The target application may integrate a connection component for connecting with the visual detection client. Specifically, the connection component may be a js-SDK plug-in.

[0119] The visual inspection client can provide a QR code for connecting to the test client. The target application integrated with the above connection component can call the camera in the above mobile phone to scan the QR code to establish a WebSocket long connection between the visual inspection client and the test client.

[0120] After the above-mentioned WebSocket persistent connection is established, the visual inspection client can have the ability to quickly obtain real-time screenshots of the application interface of the target application. The visual inspection client can call the connection component through the above-mentioned WebSocket persistent connection to obtain the interface screenshots of the application interface output to the user during the operation of the target application from the test client.

[0121] In the case where the connection is not established in the above manner, the visual inspection client may also provide a method for manually uploading a screenshot of the application interface output to the user during the operation of the target application.

[0122] See also Figure 6 , Figure 6 A schematic diagram of a visual restoration detection method according to an embodiment of the present specification is schematically shown;

[0123] After obtaining the interface screenshot, the developer can see the visual effect image and the corresponding interface screenshot in the visual inspection client.

[0124] like Figure 6 As shown, Figure 6 The visual inspection client specifically displays the visual effect pictures and interface screenshots corresponding to the "Music Tag" of the application interface in the target application.

[0125] In addition, the interface provides two specific comparison functions: coverage comparison and intelligent labeling.

[0126] The overlay comparison function can overlay the interface screenshots on the visual effect map in a semi-transparent manner, allowing developers to quickly find non-overlapping areas to optimize and correct the style.

[0127] like Figure 6 As shown, developers can adjust the overlay transparency of the interface screenshot to more clearly see non-overlapping, different areas.

[0128] Intelligent labeling allows the visual inspection client to call the Python image recognition module to identify the page elements contained in the image, and use the image comparison algorithm based on the OpenCV4.0 framework to compare the identified page elements, calculate the page elements with differences, and then mark them with red boxes.

[0129] For page elements with particularly small differences, it can be considered that they have been visually restored correctly. Therefore, a detection threshold can be set for the difference degree of the page elements.

[0130] Developers can set the sensitivity of smart annotation. Different sensitivities can correspond to different detection thresholds. Page elements on the interface screenshot whose degree of difference with the visual effect image exceeds the detection threshold corresponding to the set sensitivity will be marked, and page elements whose degree of difference with the visual effect image does not exceed the detection threshold corresponding to the set sensitivity will not be marked.

[0131] For example Figure 6 In the , the sensitivity of smart labeling can be set to "loose", "standard" and "strict" to correspond to different detection thresholds.

[0132] After marking the page elements that are different in the interface screenshot and the visual effect image, the summary information of the interface can also be displayed on the interface of the visual inspection client, including the visual restoration similarity of the application interface, the number of page elements recommended for modification, etc.

[0133] See also Figure 7 , Figure 7 A report diagram of a visual restoration detection method according to an embodiment of the present specification is schematically shown.

[0134] When one or more of the application interfaces in the target application have been intelligently labeled, a comprehensive visual restoration report for the target application can be generated.

[0135] The comprehensive report can summarize the visual restoration of all application interfaces, including indicators such as the total number of pages, detection status, restoration degree, recommended modification points, as well as materials such as original visual effect images, original interface screenshots, and annotated interface screenshots.

[0136] In an exemplary embodiment of this specification, a visual restoration detection device is also provided. Figure 8 , Figure 8 It is a block diagram of the visual restoration detection device according to the embodiment of this specification.

[0137] The device may include the following components:

[0138] The parsing unit 810 is used to obtain a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to the application interface of the target application;

[0139] A screenshot unit 820 is used to obtain a screenshot of the application interface output to the user during the operation of the target application;

[0140] The calculation unit 830 is used to call the detection algorithm on the visual detection client, calculate the page elements that are different between the interface screenshot and the visual effect image, and mark the page elements that are different on the interface screenshot.

[0141] Optionally, the target application runs on a test client;

[0142] The device also includes:

[0143] A connection unit 840, configured to establish a long connection between the visual inspection client and the test client;

[0144] The screenshot unit 820 is specifically used for:

[0145] Through the long connection, an interface screenshot of the application interface output to the user during the operation of the target application is obtained from the test client.

[0146] Optionally, the target application integrates a connection component;

[0147] The connecting unit 840 is specifically used for:

[0148] Outputting a QR code for establishing a connection, so that the test client scans the QR code to establish a WebSocket long connection between the visual inspection client and the test client;

[0149] The screenshot unit 820 is specifically used for:

[0150] The connection component is called to obtain, from the test client, a screenshot of the application interface output to the user during the running of the target application.

[0151] Optionally, the device further comprises:

[0152] The transparency unit 850 is used to make the operation screenshot transparent, and output it to the developer after superimposing it with the corresponding visual effect diagram for visual comparison by the developer.

[0153] Optionally, the calculation unit 830 is specifically configured to:

[0154] Calling an image recognition algorithm on the visual detection client to identify page elements in the interface screenshot and the visual effect image;

[0155] An image comparison algorithm on the visual detection client is called to calculate the page elements that are different between the interface screenshot and the visual effect image.

[0156] Optionally, the device further comprises:

[0157] The reporting unit 860 is used to generate a visual restoration detection report of the visual effect diagram and the visual design file; wherein the visual restoration detection report includes an interface screenshot with the page elements with differences marked.

[0158] Optionally, the visual restoration detection report also includes one or more of the following:

[0159] The visual effect diagram;

[0160] The number of page elements that differ between the interface screenshot and the visual effect diagram;

[0161] A similarity score of the interface screenshot relative to the visual effect diagram.

[0162] Optionally, the calculation unit 830 is specifically configured to:

[0163] Based on a preset detection threshold, page elements whose difference exceeds the preset detection threshold are marked on the interface screenshot.

[0164] Optionally, the visual design file includes a visual design file in Sketch format.

[0165] Optionally, the image recognition algorithm includes an image recognition algorithm based on Python;

[0166] The image comparison algorithm includes an image comparison algorithm based on the OpenCV framework.

[0167] In an exemplary embodiment of the present specification, a visual restoration detection system is also provided. Fig. 9 , Fig. 9 The structure diagram of a visual restoration detection system according to an embodiment of the present specification is schematically shown;

[0168] The system comprises:

[0169] Visual inspection client 910, used for:

[0170] Acquire a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to an application interface of the target application;

[0171] Receive a screenshot of the target application interface sent by the test client, call a detection algorithm, calculate the page elements that are different between the screenshot and the visual effect image, and mark the page elements that are different on the screenshot;

[0172] Test client 920, used for:

[0173] The target application is run, and a screenshot of the target application interface output to the user during the running of the target application is obtained and sent to the visual inspection client.

[0174] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0175] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on at least one network unit. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0176] In an exemplary embodiment of the present specification, an embodiment of a device and a terminal to which the device is applied is also provided.

[0177] The embodiments of the apparatus of this specification can be applied to computer devices, such as terminal devices. The embodiments of the apparatus can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically sensed apparatus, the processor in which the apparatus is located reads the corresponding computer program instructions in the non-volatile memory into the memory and runs the instructions. From the hardware level, such as Fig.10 As shown, Fig.10 A hardware structure diagram of a computer device 100 in which a device of an embodiment of this specification is located, except Fig.10In addition to the processor 1010, memory 1030, network interface 1020, and non-volatile memory 1040 shown, the electronic device where the device in the embodiment is located may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0178] In the exemplary embodiments of this specification, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of this specification is stored. In some possible embodiments, various aspects of this specification can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of this specification described in the above "Exemplary Method" section of this specification.

[0179] According to the embodiment of the present specification, the program product for implementing the above method can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present specification is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0180] The program product may adopt any combination of one or at least one readable medium. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or at least one wire, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0181] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0182] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0183] Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0184] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0185] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of the claimed protection, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in at least one embodiment in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in at least one embodiment or in any suitable sub-combination. In addition, although the features may work in certain combinations as described above and even initially claim protection, one or at least one feature from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0186] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring these operations to be performed in the particular order shown or to be performed sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system units and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into at least one software product.

[0187] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0188] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A visual restoration detection method, applied to a visual detection client corresponding to a developed target application; the method comprises: Acquire a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to an application interface of the target application; Obtaining a screenshot of the application interface output to the user during the operation of the target application; The detection algorithm on the visual detection client is called to calculate the page elements that are different between the interface screenshot and the visual effect image, and the page elements that are different are marked on the interface screenshot.

2. The method according to claim 1, wherein: The target application runs on a test client; The method further comprises: Establishing a persistent connection between the visual inspection client and the test client; The obtaining of a screenshot of the application interface outputted to the user during the running of the target application includes: Through the long connection, an interface screenshot of the application interface output to the user during the operation of the target application is obtained from the test client.

3. The method according to claim 2, wherein: The target application integrates a connection component; The establishing of a long connection between the visual detection client and the test client includes: Outputting a QR code for establishing a connection, so that the test client scans the QR code to establish a WebSocket long connection between the visual inspection client and the test client; The obtaining, from the test client, a screenshot of the application interface outputted to the user during the running of the target application, comprises: The connection component is called to obtain, from the test client, a screenshot of the application interface output to the user during the running of the target application.

4. The method according to claim 1, wherein: The method further comprises: The running screenshot is made transparent, and is superimposed with the corresponding visual effect diagram and then output to the developer for visual comparison.

5. The method according to claim 1, wherein: The calling of the detection algorithm on the visual detection client to calculate the page elements having differences between the interface screenshot and the visual effect image includes: Calling an image recognition algorithm on the visual detection client to identify page elements in the interface screenshot and the visual effect image; An image comparison algorithm on the visual detection client is called to calculate the page elements that are different between the interface screenshot and the visual effect image.

6. The method according to claim 1, wherein: The method further comprises: Generate a visual restoration detection report of the visual effect diagram and the visual design file; wherein the visual restoration detection report includes an interface screenshot with page elements that are different marked.

7. The method according to claim 6, wherein: The visual restoration test report may also include one or more of the following: The visual effect diagram; The number of page elements that differ between the interface screenshot and the visual effect diagram; A similarity score of the interface screenshot relative to the visual effect diagram.

8. The method according to claim 1, wherein: The marking of different page elements on the interface screenshots includes: Based on a preset detection threshold, page elements whose difference exceeds the preset detection threshold are marked on the interface screenshot.

9. The method according to claim 1, wherein: The visual design file includes a visual design file in Sketch format.

10. The method according to claim 5, wherein: The image recognition algorithm includes an image recognition algorithm based on Python; The image comparison algorithm includes an image comparison algorithm based on the OpenCV framework.

11. A visual restoration detection device, applied to a visual detection client corresponding to a developed target application; The device comprises: A parsing unit, used to obtain a visual design file corresponding to the target application, and parse the visual design file to obtain a visual effect diagram corresponding to an application interface of the target application; A screenshot unit, used to obtain a screenshot of the application interface output to the user during the operation of the target application; A calculation unit is used to call the detection algorithm on the visual detection client, calculate the page elements that are different between the interface screenshot and the visual effect image, and mark the page elements that are different on the interface screenshot.

12. A visual restoration detection system, the system comprising: Vision inspection client for: Acquire a visual design file corresponding to a target application, and parse the visual design file to obtain a visual effect diagram corresponding to an application interface of the target application; Receive a screenshot of the target application interface sent by the test client, call a detection algorithm, calculate the page elements that are different between the screenshot and the visual effect image, and mark the page elements that are different on the screenshot; Test client for: The target application is run, and a screenshot of the target application interface output to the user during the running of the target application is obtained and sent to the visual inspection client.

13. A storage medium having a computer program stored thereon, which, when executed, implements the steps of the method according to any one of claims 1 to 11.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the following method is implemented: wherein, when the processor executes the program, the method as claimed in any one of claims 1 to 11 is implemented.

Images