Image processing method and device

Abstract

The embodiment of the invention provides an image processing method and device, computer equipment, a computer readable storage medium and a computer program product, and belongs to the field of image processing. The image processing method comprises the steps of collecting a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment; on the basis of a pre-constructed conversion knowledge base and the first interface image, a multi-modal large model is used for carrying out multiple stages of processing on the second interface image so as to determine whether conversion errors exist in the second interface image or not, and the multiple stages of processing at least comprise a first round of detection and a second round of detection, the first round of detection and the second round of detection at least comprise one different detection method; and when it is determined that the conversion error exists in the second interface image, outputting information of the conversion error. According to the technical scheme provided by the embodiment of the invention, the efficiency and accuracy of conversion verification of two layouts from left to right to from right to left can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the global deployment of mobile applications and web products, the importance of software internationalization and localization testing is becoming increasingly prominent. Especially for languages ​​using right-to-left (RTL) writing conventions (such as Arabic, Hebrew, and Persian), the corresponding user interface conversion involves not only text translation but also mirroring the standard left-to-right (LTR) interface. This ensures that, while maintaining the information hierarchy and interaction logic, the converted interface appears natural, intuitive, and conforms to local reading habits for right-to-left users.

[0003] However, currently, verifying the correctness of the converted interface usually relies on manual visual inspection, which consumes a lot of manpower, time and cost, is inefficient and difficult to scale.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0005] This application provides an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems mentioned above.

[0006] One aspect of this application provides an image processing method, the method comprising: Acquire a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment; Based on a pre-built conversion knowledge base and the first interface image, a multimodal large model is used to process the second interface image in several stages to determine whether there is a conversion error in the second interface image. The processing in several stages includes at least a first round of detection and a second round of detection, and the first round of detection and the second round of detection contain at least one different detection method. If a conversion error is determined in the second interface image, the conversion error information is output.

[0007] Optionally, the acquisition of a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment includes: The basic routing table is extracted from the configuration file of the target application. The basic routing table includes the jump relationship of each user interface in the target application and the corresponding interface component. An operation decision tree is constructed based on the basic routing table. The operation decision tree is a tree-shaped logical structure with each user interface of the target application as a node and the interaction operations between the interfaces as branches. Based on the operation decision tree, a preset exploration algorithm is used to explore and collect the user interface of the target application in the left-to-right layout environment, form an exploration path and obtain the first interface image, which consists of several images. Based on the exploration path, the user interface of the target application in the right-to-left layout environment is collected to obtain the second interface image, and the number of the second interface images is the same as that of the first interface images.

[0008] Optionally, the step of exploring and collecting the user interface of the target application in the left-to-right layout environment using a preset exploration algorithm based on the operation decision tree includes: During the exploration process, obtain the current interface state of the user interface; Calculate the target interface fingerprint of the current user interface based on the interface state; The target interface fingerprint is matched with existing interface fingerprints in the interface fingerprint database, which is used to store interface fingerprints during the exploration process. If a match is successful and the current user interface has been fully explored, a backtracking of the exploration process is triggered. If a match fails, the target interface fingerprint is written into the interface fingerprint database.

[0009] Optionally, the step of exploring and collecting the user interface of the target application in the left-to-right layout environment using a preset exploration algorithm based on the operation decision tree includes: During the exploration process, the following procedure is executed repeatedly until a preset stopping condition is met: When the current user interface is a newly discovered interface, the screenshot of the current user interface and the document object model tree are input into the multimodal large model, and the multimodal large model is used to output an operation priority list, which is the operation priority of various interface components in the current interface; Based on the operation priority list, the operations of each interface component of the current user interface are placed into the execution queue; Take an operation from the queue to be executed and execute it; If the result of the operation is to enter the next interface and the next interface is a newly discovered interface, return to execute the steps of inputting the screenshot of the current user interface and the document object model tree into the multimodal large model and the following steps; If the result of the operation is that the user remains on the current user interface and all operations on the current user interface have been completed, return to the previous node of the current user interface.

[0010] Optionally, the second interface image is processed in several stages using a multimodal large model based on a pre-built conversion knowledge base and the first interface image, including: Before the first round of detection, the first interface image, the second interface image, and the corresponding document object model information are input into the multimodal large model, and the multimodal large model is used to output a structured interface description. Based on the structured interface description, the transformation knowledge base, and the first interface image, the multimodal large model is used to perform the first round of detection and the second round of detection on the second interface image, respectively, to obtain the first round of detection results and the second round of detection results; Based on the results of the first and second rounds of detection, determine whether there is a conversion error in the second interface image.

[0011] Optionally, after determining whether a conversion error exists in the second interface image based on the first round of detection results and the second round of detection results, the method further includes: Output a list of potential problems based on the conversion errors present in the second interface image; Based on the transformation knowledge base, the issues in the suspected issue list are constrained and validated, and a valid issue list is obtained based on the results of the constraint validation. The specific location coordinates of each problem in the list of valid problems are located using the multimodal large model. The second interface image is output with visual annotations based on the specific location coordinates. Correspondingly, when it is determined that there is a conversion error in the second interface image, outputting the conversion error information includes: If a conversion error is determined to exist in the second interface image, information about the conversion error is output based on the visual annotations.

[0012] Optionally, when it is determined that there is a conversion error in the second interface image, outputting the conversion error information includes: A problem list is generated based on all conversion errors in the second interface image; Determine whether the problem list contains errors that were not recorded in the conversion knowledge base; In cases involving the unrecorded error, a new case record is created based on the unrecorded error; The transformation knowledge base is updated based on the new case records.

[0013] Optionally, before performing several stages of processing on the second interface image using a multimodal large model based on a pre-built transformation knowledge base and the first interface image, the method further includes: Multi-dimensional filtering is performed on the first target image and the second target image. This multi-dimensional filtering includes at least similarity filtering and loading anomaly filtering. The loading anomaly filtering is used to filter the first target image and the second target image when the blank area of ​​either the first target image or the second target image is greater than a first threshold. The similarity filtering is performed through the following steps: The second target image is horizontally mirrored to obtain the mirrored second target image; Calculate the similarity between the mirrored second target image and the first target image; If the similarity is less than the second threshold, filter the first target image and the second target image; Wherein, the first target image is any of the first interface images, and the second target image is the second interface image corresponding to the first target image.

[0014] Another aspect of this application provides an image processing apparatus, the apparatus comprising: The acquisition module is used to acquire a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment. The processing module is used to perform several stages of processing on the second interface image based on a pre-built conversion knowledge base and the first interface image using a multimodal large model, in order to determine whether there is a conversion error in the second interface image. The several stages of processing include at least a first round of detection and a second round of detection, and the first round of detection and the second round of detection include at least one different detection method. The output module is used to output conversion error information when it is determined that there is a conversion error in the second interface image.

[0015] Another aspect of this application provides a computer device, including: At least one processor; and A memory that is communicatively connected to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0016] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0017] Another aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0018] The embodiments of this application employing the above-described technical solution may have the following advantages: By acquiring interface images from two layout environments, one from left to right and the other from right to left, a multimodal large model is used to process the interface image from right to left in several stages based on a pre-built conversion knowledge base and the interface image from left to right. The processing of these stages includes at least two rounds of detection using at least one different detection method. If the interface image from right to left is found to have a conversion error, the conversion error information is output. The multimodal large model can be constrained by the conversion knowledge base to perform multi-stage and multi-method detection and analysis on the converted user interface, automatically and accurately determining whether the conversion is correct, thus improving the efficiency and accuracy of conversion verification. Attached Figure Description

[0019] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0020] Figure 1 The diagram illustrates an example of a layout transition from left to right to right to left. Figure 2 A flowchart illustrating an image processing method according to Embodiment 1 of this application is shown schematically; Figure 3 Schematic illustration Figure 2 Flowchart of the sub-steps in step S200; Figure 4 Schematic illustration Figure 3 Flowchart of the sub-steps in step S304; Figure 5 Schematic illustration Figure 3 Flowchart of another sub-step in step S304; Figure 6 Schematic illustration Figure 2 Flowchart of the sub-steps in step S202; Figure 7 The illustration schematically shows a new flow in the image processing method according to Embodiment 1 of this application; Figure 8 Schematic illustration Figure 2 Flowchart of the sub-steps in step S204; Figure 9 This illustration schematically shows another additional process of the image processing method according to Embodiment 1 of this application; Figure 10 This schematic diagram illustrates the overall architecture of the image processing method according to Embodiment 1 of this application; Figure 11 for Figure 10 Corresponding flowchart example; Figure 12 A block diagram of an image processing apparatus according to Embodiment 2 of this application is schematically shown; and Figure 13 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0023] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0024] First, a definition of the terminology used in this application is provided: Left-to-Right (LTR): refers to languages ​​and their corresponding interface layouts where the writing and reading direction is from left to right.

[0025] Right-to-Left (RTL): refers to languages ​​where the writing and reading direction is from right to left and their corresponding interface layouts. Typical languages ​​include Arabic, Hebrew, and Persian.

[0026] Document Object Model (DOM): A programming interface for representing and manipulating the structure of HTML or XML documents.

[0027] Multimodal large model: refers to a large-scale pre-trained language model that can process multiple modal inputs (such as text, images, audio, etc.) at the same time.

[0028] Activity: In Android application development, an Activity is an application component used to host the user interface, typically corresponding to a screen or interface within the application.

[0029] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below: To adapt to right-to-left language environments, it is usually necessary to convert user interfaces (such as application user interfaces) for left-to-right speech environments so that the converted user interface conforms to the reading habits of right-to-left users, thereby improving the user experience for these users. Please refer to [reference needed]. Figure 1 The document provides an example of a transformation. To protect privacy, some areas in the image have been blurred.

[0030] To verify whether the converted interface is correct, it is usually necessary to rely on manual visual inspection one by one, which is inefficient and difficult to scale.

[0031] Among the related technologies, the following solutions also exist: 1. Structured comparison based on DOM or view hierarchy: Obtain the interface XML / HTML structure tree through an automated framework, check control attributes (such as layout direction and alignment), and determine issues such as missing components and configuration errors.

[0032] 2. Visual regression test based on image processing: By collecting screenshots of the interface to be tested and comparing them with the "gold standard image" using pixel comparison, SSIM and other algorithms, the difference is calculated to determine whether the test passes.

[0033] 3. OCR-based text verification technology: Extract text from screenshots using an OCR engine, compare it with a multilingual resource library, and detect issues such as untranslated text and text errors.

[0034] 4. Single-round analysis based on a general multimodal large model: Input screenshots into the multimodal large model, and use single-round prompts to allow the model to describe the interface, identify problems, and summarize differences.

[0035] However, the above methods have the following drawbacks: 1. Structured comparison schemes based on DOM or view hierarchy are difficult to infer from the hierarchy whether the user interface truly achieves the correct visual mirroring in a right-to-left environment. Problems such as custom view drawing errors, lack of mirrored versions of image resources, line height collapse due to missing font files, text and icon overlap, text truncation, and component occlusion may occur.

[0036] 2. The visual regression testing solution based on image processing will judge normal mirror changes from right to left as differences, resulting in a high false alarm rate and requiring a large amount of manual screening. When the product is frequently iterated, a large number of benchmark screenshots need to be maintained separately for each right-to-left language, which is costly.

[0037] 3. OCR-based text verification technology only focuses on whether the text is translated correctly, without paying attention to core issues such as layout errors and improper mirroring; moreover, in scenarios that include both system user interfaces and user-generated content, recognition errors and noise are prone to occur.

[0038] 4. The single-round analysis scheme based on a general multimodal large model has high complexity, unstable model output, and low structure due to the high burden of multiple tasks on the single-round prompts. In addition, it lacks the constraint of knowledge corresponding to the conversion between two language environments, which is prone to misjudgment. Furthermore, it has no caching mechanism, and repeated calls to the model result in high inference cost and low throughput.

[0039] Therefore, this application provides an image processing technical solution. In this solution, a multimodal large model is used to process the user interface of two language environments in multiple stages based on a pre-built conversion knowledge base. These multiple stages include at least a first round of detection and a second round of detection, each containing at least one different detection method. Finally, conversion error information is output. The conversion knowledge base can constrain the multimodal large model to perform multi-stage and multi-method detection and analysis on the converted user interface, automatically and accurately determining whether the conversion is correct, thus improving the efficiency and accuracy of conversion verification. See below for details.

[0040] The technical solutions of this application are described below through several embodiments. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0041] Example 1 Figure 2A flowchart illustrating an image processing method according to Embodiment 1 of this application is shown. It should be noted that the execution entity of the image processing method in this embodiment can be a server or a client; the following description uses a server as the execution entity.

[0042] like Figure 2 As shown, the image processing method may include steps S200~S204, wherein: Step S200: Acquire a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment.

[0043] Step S202: Based on the pre-built conversion knowledge base and the first interface image, a multimodal large model is used to process the second interface image in several stages to determine whether there is a conversion error in the second interface image. The processing in several stages includes at least a first round of detection and a second round of detection, and the first round of detection and the second round of detection contain at least one different detection method.

[0044] Step S204: If it is determined that there is a conversion error in the second interface image, output the conversion error information.

[0045] The image processing method provided in this embodiment acquires interface images of two layout environments: left-to-right and right-to-left. Based on a pre-built conversion knowledge base and the left-to-right interface image, a multimodal large model is used to process the right-to-left interface image in several stages. The processing in several stages includes at least two rounds of detection using at least one different detection method. If it is determined that the right-to-left interface image stores a conversion error, the conversion error information is output. The conversion knowledge base can constrain the multimodal large model to perform multi-stage and multi-method detection and analysis on the converted user interface, automatically and accurately determining whether the conversion is correct, thus improving the efficiency and accuracy of conversion verification.

[0046] The following combination Figure 2 The steps in steps S200 to S204, as well as other optional steps, are described in detail.

[0047] Step S200 The system collects a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment.

[0048] The first interface image can be the user interface of an application. The second interface image is the interface image after the first interface image has been transformed. Generally, the language of the second interface image is different from that of the first interface image; that is, the transformation of the second interface image relative to the first interface image includes not only a change in interface layout but also a change in language. For example, the language corresponding to the first interface image can be Chinese, while the language corresponding to the second interface image can be Arabic.

[0049] It is understandable that an application generally includes multiple user interfaces, such as a homepage, settings page, and personal center page. Correspondingly, there can be multiple first interface images and multiple second interface images, and there is a one-to-one correspondence between the multiple first interface images and the multiple second interface images.

[0050] When acquiring the first and second interface images, screenshots of the corresponding user interfaces can be taken, or image processing tools can be used to obtain them.

[0051] Step S202 Based on a pre-built conversion knowledge base and a first interface image, a multimodal large model is used to process the second interface image in several stages to determine whether there is a conversion error in the second interface image. The processing in several stages includes at least a first round of detection and a second round of detection, and the first round of detection and the second round of detection contain at least one different detection method.

[0052] The knowledge base transformation can include the following three parts: 1. Conversion Standards: Define user interface adaptation standards for left-to-right and right-to-left environments. For example, text should be aligned from right to left; the order of child elements in a horizontal linear layout (Row) should be reversed; directional icons such as navigation bar back buttons and progress indicators need to be mirrored, while icons that convey meaning through their shape (such as play buttons and checkboxes) usually do not need to be mirrored.

[0053] 2. Typical Error Cases: This section compiles frequently occurring defect patterns from historical regression testing. These include, but are not limited to: text obscuring due to layout not being flipped, incorrect icon orientation (e.g., the "right arrow" still points to the right when viewed from right to left), disordered display of Arabic numerals and Hebrew text when mixed, and incorrect physical button mapping orientation.

[0054] 3. Exception Rules: Record special scenarios that do not follow the regular mirroring logic. For example, the progress bar of a media player usually remains from left to right; phone numbers, URLs, code snippets, etc., should still maintain the left-to-right reading order in right-to-left mode; and some brand logos or proper nouns are not allowed to be mirrored.

[0055] It should be understood that the above are merely examples of the content that can be included in the knowledge base transformation, and are not intended to limit the content that can be included in the knowledge base transformation.

[0056] Specifically, a transformation knowledge base, a first interface image, and a second interface image can be input into a multimodal large model. Cue word engineering guides the multimodal large model to perform multi-stage processing on the second interface image based on the transformation knowledge base and the first interface image, analyzing and determining whether transformation errors exist in the second interface image. The cue word engineering for each stage can be different. When there are multiple first and second interface images, each input to the multimodal large model is a corresponding pair of images. Furthermore, in the multi-stage processing, the output of an earlier stage can be used as the input for a later stage. The first round of detection and the second round of detection can belong to the same stage or different stages. The difference between the detection methods in the first and second rounds of detection can be due to differences in the prompt word engineering. For example, the prompt for the first round of detection could be: "As a test expert for the application's user interface, traverse all areas of the first and second interface images to find all possible locations with mirroring errors, layout deviations, and translation problems," to guide the multimodal large model to traverse all areas. The prompt for the second round of detection could be: "As a right-to-left interface adaptation expert, focus on checking whether directional words in the second interface image have semantic inversion, whether directional icons are mirrored, and whether mixed text (such as numbers + local text + foreign text) is correctly sorted," to guide the multimodal large model to focus on specific areas for detection. Optionally, the first and second rounds of detection can differ in their focus on the input data, the parameters used in the model, and the application of exception rules. For example, the focus on the input data could be different if the knowledge base includes two sub-bases with different focuses (e.g., one sub-base focuses on layout specifications, and the other on translation rules), with each round using one sub-base. The parameters used in the model could also differ, for example, if the first round uses a fast inference mode with lower inference accuracy, and the second round uses a high-precision inference mode with higher inference accuracy. The application of exception rules could also differ, for example, if the first round does not inform the multimodal large model of exception scenarios in advance (e.g., brand logos are not mirrored, progress bars remain from left to right), while the second round informs the multimodal large model of these exception scenarios in advance.

[0057] Understandably, the first and second rounds of detection are the main analytical processes for confirming whether there are conversion errors in the second interface image. In addition, the multi-stage processing can include other stages, such as auxiliary detection stages or stages for further screening, which are not limited here. Of course, in some scenarios, two-round detection can be simplified to single-round detection, such as in some low-computing-power scenarios; while in other scenarios, two-round detection can be extended to three or more rounds of multi-view calibration (such as focusing on layout, translation, interaction, etc.). In practical applications, expert model combinations can be used to implement two or more rounds of detection. Each round of detection can call multimodal large models with different architectures or focusing on different domains (such as one good at layout analysis and another good at text semantics), and finally the results are summarized, rather than simply changing the prompts to achieve differentiated detection. Furthermore, the first and second rounds of detection can be executed in parallel. In addition, for some high-risk user interfaces, a manual review step can be added.

[0058] In some possible embodiments, the input to the multimodal large model may also include the design draft corresponding to the interface, a standard left-to-right baseline diagram, etc., to assist the multimodal large model in making more accurate difference comparisons.

[0059] Step S204 If a conversion error is detected in the second interface image, output the conversion error information.

[0060] If a conversion error is detected in the second interface image, a multimodal large model can be used to output specific conversion error information, such as "the number direction should remain from left to right." If multiple errors exist in the second interface image, the output information can be a list of problems. To make it easier for users to understand the specific error locations in the second interface image, these locations can be marked, for example, by circling the conversion error in red.

[0061] In an optional embodiment, in step S200, a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment are acquired, such as... Figure 3 As shown, it may further include: Step S300: Extract the basic routing table based on the target application's configuration file. The basic routing table includes the navigation relationships of each user interface in the target application and the corresponding interface components.

[0062] Step S302: Construct an operation decision tree based on the basic routing table. The operation decision tree is a tree-shaped logical structure with the target application's interface as nodes and the interaction operations between interfaces as branches.

[0063] Step S304: Based on the operation decision tree, a preset exploration algorithm is used to explore and collect the user interface of the target application in the left-to-right layout environment, form an exploration path and obtain a first interface image, which consists of several images.

[0064] Step S306: Based on the exploration path, collect the user interface of the target application in the right-to-left layout environment to obtain a second interface image. The number of second interface images is the same as that of the first interface images.

[0065] Configuration files can include files containing routing information for the target application, such as AndroidManifest.xml for Android applications, routing framework registration code, etc. Extract the basic routing table from these configuration files, which contains the navigation relationships of various user interfaces (Activities) in the target application and the corresponding UI components (such as buttons) for navigation.

[0066] After extracting the basic routing table, we can construct an operation decision tree by using each user interface in the basic routing table as a node and the interactions between these user interfaces as branches. For example, we can use the homepage as the root node, and the settings page as a child node of the homepage, with the corresponding UI component being the settings button on the homepage… After constructing the operation decision tree, a preset exploration algorithm can be used to explore the user interface of the target application in a left-to-right layout environment based on the "map" indicated by the operation decision tree. Whenever a new user interface is encountered, it is captured as the first interface image, and a corresponding exploration path is formed. The exploration path can be, for example: Homepage – Settings Page – Personal Center Page. The preset exploration algorithm can be Depth-First Search (DFS), Breadth-First Search (BFS), or heuristic search. Preferably, the preset exploration algorithm is Depth-First Search.

[0067] After all user interfaces in the left-to-right layout environment have been captured or the exploration has reached a certain coverage (e.g., 90%), a complete exploration path can be obtained. Then, user interfaces in the right-to-left layout environment can be captured based on this complete exploration path to obtain the second interface image.

[0068] In this embodiment, a basic routing table is extracted based on the target application's configuration file. An operation decision tree is constructed based on the basic routing table. A preset exploration algorithm is used to explore and collect the user interface of the target application in a left-to-right layout environment based on the operation decision tree, forming an exploration path and obtaining a first interface image. Based on the exploration path, the user interface of the target application in a right-to-left layout environment is collected to obtain a second interface image. The same path can be used to collect the first and second interface images with the same interface state, the same number, and the same correspondence, which facilitates conversion verification and improves the efficiency of conversion verification.

[0069] In an optional embodiment, in step S304, a preset exploration algorithm is used based on the operation decision tree to explore and collect the user interface of the target application in a left-to-right layout environment, such as... Figure 4 As shown, it may further include: Step S400: During the exploration process, obtain the current interface state of the user interface.

[0070] Step S402: Calculate the target interface fingerprint of the current user interface based on the interface state.

[0071] Step S404: Match the target interface fingerprint with the existing interface fingerprints in the interface fingerprint database, which is used to store interface fingerprints during the exploration process.

[0072] Step S406: If a match is successful and the current user interface has been explored, trigger a backtracking of the exploration process.

[0073] In step S408, if the matching fails, the target interface fingerprint is written into the interface fingerprint database.

[0074] Before exploration begins, the state trees of each user interface in the operation decision tree can be initialized. During exploration, whenever a user interface is explored, its current state is obtained, such as the Activity name and Document Object Model. Based on these states, the target interface fingerprint is calculated. Then, the target interface fingerprint is matched with existing interface fingerprints in the interface fingerprint database. If a match is successful, it means the current user interface has already been explored. In this case, further judgment can be made as to whether the current user interface has been fully explored. If it has, backtracking of the exploration process is triggered, for example, backtracking to the previous node of the current user interface. If it has not been fully explored, the exploration of the current user interface can continue. If a match fails, it means the current user interface has not been explored before, and the target interface fingerprint is written to the interface fingerprint database. For example, when exploring the first user interface, the interface fingerprint database is empty, so a match will inevitably fail, and the fingerprint will be written to the database. Or, when backtracking is triggered, the current user interface has already been explored, so a match will succeed. The interface fingerprint database can be built before exploration, meaning a new database can be created before exploration to store the calculated interface fingerprints. Additionally, "the current user interface has been explored" can mean that all interactive interface components have been explored.

[0075] In this embodiment, by calculating the target interface fingerprint of the current user interface during the exploration process, the target interface fingerprint is matched with the existing interface fingerprints in the interface fingerprint database to trigger backtracking or to write the target interface fingerprint into the interface fingerprint database. This can accurately identify duplicate interfaces, prevent the exploration process from getting stuck in an invalid loop, and improve the efficiency of the exploration.

[0076] In an optional embodiment, in step S304, exploring and collecting the user interface of the target application in a left-to-right layout environment using a preset exploration algorithm based on the operation decision tree may include: during the exploration process, repeatedly executing, as follows Figure 5 The steps shown continue until a preset stopping condition is met, where: Step S500: If the current user interface is a newly discovered interface, input the screenshot of the current user interface and the document object model tree into the multimodal large model, and use the multimodal large model to output the operation priority list, which is the operation priority of various interface components in the current interface.

[0077] Step S502: Based on the operation priority list, put the operations of each interface component of the current user interface into the execution queue.

[0078] Step S504: Take an operation from the queue to be executed and execute it.

[0079] Step S506: If the result of the operation is to enter the next interface and the next interface is a newly discovered interface, return to execute the steps of inputting the screenshot of the current user interface and the document object model tree into the multimodal large model and the following steps.

[0080] Step S508: If the result of the operation is that the user remains in the current user interface and all operations in the current user interface have been completed, return to the parent node of the current user interface.

[0081] The preset stopping condition can be reaching a predetermined exploration coverage rate or all user interfaces have been explored.

[0082] Specifically, during the exploration process, when a user interface is discovered, it can be judged to determine whether it is a newly discovered interface. For example, the interface fingerprint from the previous embodiment can be used for judgment. If a match fails, the current user interface is determined to be a newly discovered interface. At this time, a screenshot of the current user interface can be taken, and the screenshot and the document object model of the current user interface can be input into the multimodal large model. The multimodal large model can then output an operation priority list. Prompt word engineering can be used to guide the multimodal large model in outputting the operation priority list. For example, the prompt could be: "Please output the operation priority list of the current user interface in the following priority order: navigation components (entering a new user interface) → business function components (such as pop-ups, forms, etc.) → pure display components." The following is an output example: [{"action":"click","target":"btn_settings","priority":1},{"action":"click","target":"btn_profile","priority":2}] Among them, the field "action" is the type of interaction action to be executed, and the field value is "click". The field "target" is the identifier of the interaction target, and the field value is "btn_settings (settings button)" and "btn_profile (personal homepage button)". The field "priority" is the operation priority, and the field value is the corresponding priority value.

[0083] After obtaining the operation priority list of the current user interface, the operations of each interface component in the current user interface are placed into a pre-created execution queue according to the operation priority list; during exploration, an operation is taken from the execution queue and executed; if the execution result of the operation is to enter the next interface and the next interface is a newly discovered interface, return to the execution step S500 and subsequent steps; if the execution result of the operation is to stay in the current user interface and all operations of the current user interface have been completed, return to the parent node of the current user interface.

[0084] In this embodiment, by using a multimodal large model to output an operation priority list for the newly invented user interface, and placing various operations of the user interface into a queue to be executed according to the operation priority list, and retrieving operations from the queue to be executed, the exploration process can be carried out in a certain priority order, ensuring that the entire exploration process is carried out efficiently, prioritizing the coverage of core interfaces and key operations, reducing ineffective exploration, and improving the efficiency of exploration.

[0085] In an optional embodiment, in step S202, the second interface image is processed in several stages using a multimodal large model based on a pre-built conversion knowledge base and the first interface image, such as... Figure 6 As shown, it may further include: Step S600: Before the first round of detection, the first interface image, the second interface image, and the corresponding document object model information are input into the multimodal large model, and the multimodal large model is used to output a structured interface description.

[0086] Step S602: Based on the structured interface description, the transformation knowledge base, and the first interface image, a multimodal large model is used to perform a first round of detection and a second round of detection on the second interface image, respectively, to obtain the first round of detection results and the second round of detection results.

[0087] Step S604: Determine whether there is a conversion error in the second interface image based on the results of the first and second rounds of detection.

[0088] Specifically, before the first round of detection, the first interface image, the second interface image, and the corresponding document object model (Document Object Model) for the first and second interface images are input into the multimodal large model. The multimodal large model then outputs structured interface descriptions of the first and second interface images. The output of these structured interface descriptions can be guided by prompting techniques, such as: "As an application user interface analysis expert, describe the main functions and overall layout structure of the interface in natural language, list the areas that need to be checked for mirror adaptation, such as the navigation bar, operation button rows, list area, and tab bar, and explain the expected performance of each area in a right-to-left layout environment." The output structured interface descriptions can be in JSON format for easy use later. Furthermore, the first and second interface images can be stitched together into a single image before being input into the multimodal large model, or they can be left unstitched.

[0089] It is understandable that the process of using a multimodal large model to output a structured interface description can be considered as one of several stages.

[0090] After obtaining the structured interface description, a first round of detection and a second round of detection can be performed using this description. The inputs for both rounds can be the same: the structured interface description, the transformation knowledge base, the first interface image, and the second interface image. In this case, the first and second rounds of detection can use differentiated prompts to achieve different detection methods. For example, the prompts for the first round could instruct the multimodal large model to find all regions that may have mirroring, layout, or translation problems, emphasizing "better to overreport than miss." The prompts for the second round could be differentiated by changing the order of focus or adjusting the wording, aiming to cover as many problems as possible that the first round might have missed. Optionally, in the first and second rounds of detection, the multimodal large model can perform the following checks on translation quality: 1. Check for untranslated or missing translations: Check if any source language text remains in the second interface image; 2. Check if the text is truncated or overflows: Check if the image on the second interface exceeds the control boundary due to increased length, resulting in incomplete display or overlap; 3. Are there any directional vocabulary errors? Check whether the directional words (such as "swipe left", "right side", etc.) contained in the second interface image have been semantically reversed according to the right-to-left layout; 4. Check for placeholder leaks: Check if code placeholders such as %s and {name} are directly displayed in the second interface image.

[0091] By conducting the first and second rounds of detection, we can obtain the results of the first and second rounds of detection. Then, based on the results of the two rounds of detection, we can determine whether there are any conversion errors in the second interface image.

[0092] In this embodiment, by utilizing the structured interface description output by the multimodal large model, and then performing two rounds of detection based on the structured interface description using the multimodal large model, the multimodal large model can perform two rounds of detection in a targeted manner according to the structured interface description, thereby improving the efficiency and accuracy of the two rounds of detection.

[0093] In an optional embodiment, after step S604, that is, after determining whether there is a conversion error in the second interface image based on the first round of detection results and the second round of detection results, such as Figure 7 As shown, the image processing method in this embodiment may further include: Step S700: Output a list of suspected problems based on the conversion errors present in the second interface image.

[0094] Step S702: Perform constraint verification on the issues in the suspected issue list based on the transformation knowledge base, and obtain the valid issue list based on the result of the constraint verification.

[0095] Step S704: Use the multimodal large model to locate the specific coordinates of each problem in the list of valid problems.

[0096] Step S706: Output the visual annotation of the second interface image based on the specific location coordinates.

[0097] Specifically, the results of the first and second rounds of detection can be merged according to the location and type of the problem, and duplicates can be removed to form a list of suspected problems. Optionally, each problem record may include fields such as: location description, problem type (mirror, layout, or translation), comparison of the states on the left and right sides, description of the suspected problem, and confidence level.

[0098] Then, each suspected problem in the suspected problem list can be compared with the conversion specifications and typical error cases in the conversion knowledge base for constraint verification. For translation problems, the terminology or non-translation word list (such as brand names) in the conversion knowledge base can be referenced to prevent normal reserved words from being misjudged as omitted translations. After the constraint verification is completed, a list of valid problems is formed based on the results of the constraint verification. For each valid problem, the severity level of the problem can be given, such as high, medium, or low, and a final description of the problem can also be provided.

[0099] For each valid question in the valid question list, a multimodal large model can be used to locate each valid question, obtaining the precise location coordinates corresponding to each finite question, such as the coordinates of the top left and top right corners of the bounding box, or the percentage position relative to the image size. Optionally, the specific location coordinates of each question can be placed in the finite question list.

[0100] After obtaining the specific location coordinates of each problem, visual annotations can be drawn on the stitched image or the second interface image based on these specific location coordinates. Correspondingly, in step S204, if it is determined that there is a conversion error in the second interface image, outputting the conversion error information may include: outputting the conversion error information based on the visual annotations if it is determined that there is a conversion error in the second interface image.

[0101] Optionally, the complete analysis results of each image pair consisting of a first interface image and a second interface image can be summarized into a structured record. This structured record may include: interface identifier, stitched image path, standard image path, analysis details for each stage, a list of valid issues, and final conclusions (e.g., "2 layout issues and 1 translation issue were found"). The structured record can be written to the output directory in JSON format. When provided to testers, the results file and annotated screenshots can be loaded using a web viewer module, and a visualization interface can be provided via a local HTTP service.

[0102] In this embodiment, the list of suspected problems output from two rounds of detection is constrained and validated using a transformation knowledge base to obtain a list of valid problems. Based on the specific location coordinates of the valid problems, a visual annotation of the second interface image is output. Subsequently, information on transformation errors is output based on the visual annotation. The problem can be further confirmed through constraint validation using the transformation knowledge base, thereby improving the accuracy of the output analysis results. At the same time, the visual annotation makes it convenient for users to view and confirm the transformation errors present in the second interface image.

[0103] In an optional embodiment, in step S204, if it is determined that there is a conversion error in the second interface image, information about the conversion error is output, such as... Figure 8 As shown, it may further include: Step S800: A problem list is generated based on all conversion errors of the second interface image.

[0104] Step S802: Determine whether the problem list contains errors related to unrecorded knowledge base conversion.

[0105] Step S804: In cases where there are unrecorded errors, a new case record is generated based on the unrecorded errors.

[0106] Step S806: Update the transformation knowledge base based on the new case records.

[0107] After several stages of processing, conversion errors in the second interface image can be identified, and a problem list is formed based on all existing conversion errors. Alternatively, the valid problem list from the previous embodiment can be used directly. Then, each problem in the problem list can be iterated through to confirm whether there are any errors not recorded in the conversion knowledge base. For example, each problem in the problem list can be compared with errors in typical error cases to confirm whether the current problem is already recorded in the typical error cases. If the current problem is not recorded there, a new case record can be formed based on the current problem, and the new case record can be updated in the conversion knowledge base, such as updating it in the typical error cases.

[0108] In this embodiment, a problem list is formed based on all conversion errors of the second interface image. It is then determined whether the problem list contains errors not recorded in the conversion knowledge base. If such errors are found, a new case record is created to update the conversion knowledge base, which can continuously enrich the conversion knowledge base and thus continuously improve the accuracy of conversion verification.

[0109] In an optional embodiment, before step 202, before recording the processing of the second interface image using a multimodal large model based on a pre-built conversion knowledge base and the first interface image, the image processing method of this embodiment may further include: performing multi-dimensional filtering on the first target image and the second target image. The multi-dimensional filtering includes at least similarity filtering and loading anomaly filtering. Loading anomaly filtering is used to filter the first target image and the second target image when the blank area of ​​either the first target image or the second target image is greater than a first threshold. The first target image is any first interface image, and the second target image is the second interface image corresponding to the first target image. Similarity filtering is performed through methods such as... Figure 9 The steps shown are for filtering, where: Step S900: Mirror the second target image horizontally to obtain the mirrored second target image.

[0110] Step S902: Calculate the similarity between the mirrored second target image and the first target image.

[0111] Step S904: If the similarity is less than the second threshold, filter the first target image and the second target image.

[0112] Specifically, in the loading anomaly filtering, blank areas can be detected in both the first and second target images. If the blank area is larger than a first threshold (e.g., 90%), it indicates that the image is loading abnormally and needs to be filtered. The first threshold can be set according to the actual situation.

[0113] In similarity filtering, the second target image can be horizontally mirrored to obtain the mirrored second target image. Then, the similarity between the mirrored second target image and the first target image is calculated. If the similarity is less than a second threshold (e.g., 50%), it means that the similarity is too low and the first and second target images need to be filtered.

[0114] In practical applications, in addition to the above-mentioned filtering, other filtering can also be included, such as filtering based on OCR text density (e.g., interfaces with very little text are judged as abnormal), filtering based on the depth of the document object model tree (e.g., interfaces with too few nodes are judged as loading failure), etc. There are no restrictions here. In addition, other preprocessing can be performed on the first target image and the second target image, such as cropping the top 5% of the screenshot to remove noise unrelated to the application's user interface, such as status bar and system notifications.

[0115] In this embodiment, by performing multi-dimensional filtering on the first target image and the second target image, including at least similarity filtering and loading anomaly filtering, images that do not meet the processing requirements can be filtered out, ineffective conversion verification processing can be avoided, and the utilization rate of conversion verification computing resources and the accuracy of conversion verification can be improved.

[0116] To make this application easier to understand, the following is combined with... Figure 10 and Figure 11 An example application is provided.

[0117] Please refer to Figure 10 The figure shows the overall architecture of the image processing method according to an embodiment of this application. As shown, it can be divided into the following logical modules: (1) Automated Screenshot Acquisition Module: This module is responsible for automatically traversing all pages within the application and collecting screenshots in both left-to-right and right-to-left language / layout environments, providing input data for subsequent comparative analysis. This module employs a dual-track strategy combining "route-driven + intelligent exploration" to maximize page coverage. It outputs paired LTR / RTL screenshot directories and corresponding Document Object Model (DOM) information directories.

[0118] (2) Intelligent preprocessing and filtering module: preprocesses and filters the input screenshots, including mirror similarity filtering, blank area detection, top cropping, size alignment and splicing, and outputs a list of filtered valid tasks.

[0119] (3) Four-stage multimodal analysis module: The core module of the system, consisting of four interconnected analysis stages—the first stage (page description and comparison of key identification), the second stage (dual-wheel error detection and merging), the third stage (strict screening and location), and the fourth stage (knowledge base update and error annotation).

[0120] (4) Results output and visualization module: Generates annotated screenshots and structured JSON reports, and provides a web visualization interface for testers to view and filter issues.

[0121] Figure 11 To and Figure 10 The corresponding implementation process generally includes the following: 1. Capture user interface images in both left-to-right and right-to-left layout environments; 2. Based on the application's configuration file, a basic routing table is proposed to form an operation decision tree. An exploration algorithm is used to explore and collect user interface images from the left to right layout environment to form an exploration path. 3. Collect user interface images from a right-to-left layout environment based on the exploration path; 4. Perform similarity filtering, white space filtering, size cropping, and stitching on the user interface images collected in the two layout environments; 5. Input the processed stitched image into the multimodal large model, and use the multimodal large model and knowledge base to perform analysis and processing in four stages: page description and comparison of key points identification, dual-round error detection and problem merging, strict screening and precise location, and knowledge base update and error annotation. 6. Output the analysis and processing results.

[0122] In this exemplary application, by utilizing a multimodal large model to perform four-stage analysis and processing on the user interfaces of the two language environments, it is possible to automatically and accurately determine whether the conversion is correct, thereby improving the efficiency and accuracy of conversion verification.

[0123] Example 2 Figure 12 The diagram schematically illustrates a block diagram of an image processing apparatus according to Embodiment 2 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 12 As shown, the device 1000 may include: a data acquisition module 1010, a processing module 1020, and an output module 1030, wherein: The acquisition module 1010 is used to acquire a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment. The processing module 1020 is used to perform several stages of processing on the second interface image based on a pre-built conversion knowledge base and the first interface image using a multimodal large model, in order to determine whether there is a conversion error in the second interface image. The several stages of processing include at least a first round of detection and a second round of detection, and the first round of detection and the second round of detection include at least one different detection method. The output module 1030 is used to output conversion error information when it is determined that there is a conversion error in the second interface image.

[0124] In an optional embodiment, the acquisition module 1010 is further configured to: The basic routing table is extracted from the configuration file of the target application. The basic routing table includes the jump relationship of each user interface in the target application and the corresponding interface component. An operation decision tree is constructed based on the basic routing table. The operation decision tree is a tree-shaped logical structure with each user interface of the target application as a node and the interaction operations between the interfaces as branches. Based on the operation decision tree, a preset exploration algorithm is used to explore and collect the user interface of the target application in the left-to-right layout environment, form an exploration path and obtain the first interface image, which consists of several images. Based on the exploration path, the user interface of the target application in the right-to-left layout environment is collected to obtain the second interface image, and the number of the second interface images is the same as that of the first interface images.

[0125] In an optional embodiment, the acquisition module 1010 is further configured to: During the exploration process, obtain the current interface state of the user interface; Calculate the target interface fingerprint of the current user interface based on the interface state; The target interface fingerprint is matched with existing interface fingerprints in the interface fingerprint database, which is used to store interface fingerprints during the exploration process. If a match is successful and the current user interface has been fully explored, a backtracking of the exploration process is triggered. If a match fails, the target interface fingerprint is written into the interface fingerprint database.

[0126] In an optional embodiment, the acquisition module 1010 is further configured to: During the exploration process, the following procedure is executed repeatedly until a preset stopping condition is met: When the current user interface is a newly discovered interface, the screenshot of the current user interface and the document object model tree are input into the multimodal large model, and the multimodal large model is used to output an operation priority list, which is the operation priority of various interface components in the current interface; Based on the operation priority list, the operations of each interface component of the current user interface are placed into the execution queue; Take an operation from the queue to be executed and execute it; If the result of the operation is to enter the next interface and the next interface is a newly discovered interface, return to execute the steps of inputting the screenshot of the current user interface and the document object model tree into the multimodal large model and the following steps; If the result of the operation is that the user remains on the current user interface and all operations on the current user interface have been completed, return to the previous node of the current user interface.

[0127] In an optional embodiment, the processing module 1020 is further configured to: Before the first round of detection, the first interface image, the second interface image, and the corresponding document object model information are input into the multimodal large model, and the multimodal large model is used to output a structured interface description. Based on the structured interface description, the transformation knowledge base, and the first interface image, the multimodal large model is used to perform the first round of detection and the second round of detection on the second interface image, respectively, to obtain the first round of detection results and the second round of detection results; Based on the results of the first and second rounds of detection, determine whether there is a conversion error in the second interface image.

[0128] In an optional embodiment, the device 1000 is further used for: Output a list of potential problems based on the conversion errors present in the second interface image; Based on the transformation knowledge base, the issues in the suspected issue list are constrained and validated, and a valid issue list is obtained based on the results of the constraint validation. The specific location coordinates of each problem in the list of valid problems are located using the multimodal large model. The second interface image is output with visual annotations based on the specific location coordinates. Correspondingly, the output module 1030 is also used for: If a conversion error is determined to exist in the second interface image, information about the conversion error is output based on the visual annotations.

[0129] In an optional embodiment, the output module 1030 is further configured to: A problem list is generated based on all conversion errors in the second interface image; Determine whether the problem list contains errors that were not recorded in the conversion knowledge base; In cases involving the unrecorded error, a new case record is created based on the unrecorded error; The transformation knowledge base is updated based on the new case records.

[0130] In an optional embodiment, the device 1000 is further used for: Multi-dimensional filtering is performed on the first target image and the second target image. This multi-dimensional filtering includes at least similarity filtering and loading anomaly filtering. The loading anomaly filtering is used to filter the first target image and the second target image when the blank area of ​​either the first target image or the second target image is greater than a first threshold. The similarity filtering is performed through the following steps: The second target image is horizontally mirrored to obtain the mirrored second target image; Calculate the similarity between the mirrored second target image and the first target image; If the similarity is less than the second threshold, filter the first target image and the second target image; Wherein, the first target image is any of the first interface images, and the second target image is the second interface image corresponding to the first target image.

[0131] Example 3 Figure 13 This illustration schematically depicts the hardware architecture of a computer device 10000 suitable for implementing an image processing method according to Embodiment 3 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, or robot. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 13 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein: The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for image processing methods. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0132] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0133] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0134] It should be pointed out that, Figure 13 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0135] In this embodiment, the image processing method stored in memory 10010 can be further divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiments of this application.

[0136] Example 4 This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the image processing method in the embodiments.

[0137] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium may include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the image processing method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0138] Example 5 This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0139] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0140] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment; Based on a pre-built conversion knowledge base and the first interface image, a multimodal large model is used to process the second interface image in several stages to determine whether there is a conversion error in the second interface image. The processing in several stages includes at least a first round of detection and a second round of detection, and the first round of detection and the second round of detection contain at least one different detection method. If a conversion error is determined in the second interface image, the conversion error information is output.

2. The method according to claim 1, characterized in that, The acquisition of a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment includes: The basic routing table is extracted from the configuration file of the target application. The basic routing table includes the jump relationship of each user interface in the target application and the corresponding interface component. An operation decision tree is constructed based on the basic routing table. The operation decision tree is a tree-shaped logical structure with each user interface of the target application as a node and the interaction operations between the interfaces as branches. Based on the operation decision tree, a preset exploration algorithm is used to explore and collect the user interface of the target application in the left-to-right layout environment, form an exploration path and obtain the first interface image, which consists of several images. Based on the exploration path, the user interface of the target application in the right-to-left layout environment is collected to obtain the second interface image, and the number of the second interface images is the same as that of the first interface images.

3. The method according to claim 2, characterized in that, The process of exploring and collecting user interfaces of the target application in the left-to-right layout environment using a preset exploration algorithm based on the operation decision tree includes: During the exploration process, obtain the current interface state of the user interface; Calculate the target interface fingerprint of the current user interface based on the interface state; The target interface fingerprint is matched with existing interface fingerprints in the interface fingerprint database, which is used to store interface fingerprints during the exploration process. If a match is successful and the current user interface has been fully explored, a backtracking of the exploration process is triggered. If a match fails, the target interface fingerprint is written into the interface fingerprint database.

4. The method according to claim 2, characterized in that, The process of exploring and collecting user interfaces of the target application in the left-to-right layout environment using a preset exploration algorithm based on the operation decision tree includes: During the exploration process, the following procedure is executed repeatedly until a preset stopping condition is met: When the current user interface is a newly discovered interface, the screenshot of the current user interface and the document object model tree are input into the multimodal large model, and the multimodal large model is used to output an operation priority list, which is the operation priority of various interface components in the current interface; Based on the operation priority list, the operations of each interface component of the current user interface are placed into the execution queue; Take an operation from the queue to be executed and execute it; If the result of the operation is to enter the next interface and the next interface is a newly discovered interface, return to execute the steps of inputting the screenshot of the current user interface and the document object model tree into the multimodal large model and the following steps; If the result of the operation is that the user remains on the current user interface and all operations on the current user interface have been completed, return to the previous node of the current user interface.

5. The method according to claim 1, characterized in that, The process of processing the second interface image using a multimodal large model based on a pre-built conversion knowledge base and the first interface image includes: Before the first round of detection, the first interface image, the second interface image, and the corresponding document object model information are input into the multimodal large model, and the multimodal large model is used to output a structured interface description. Based on the structured interface description, the transformation knowledge base, and the first interface image, the multimodal large model is used to perform the first round of detection and the second round of detection on the second interface image, respectively, to obtain the first round of detection results and the second round of detection results; Based on the results of the first and second rounds of detection, determine whether there is a conversion error in the second interface image.

6. The method according to claim 5, characterized in that, After determining whether the second interface image has a conversion error based on the first round of detection results and the second round of detection results, the method further includes: Output a list of potential problems based on the conversion errors present in the second interface image; Based on the transformation knowledge base, the issues in the suspected issue list are constrained and validated, and a valid issue list is obtained based on the results of the constraint validation. The specific location coordinates of each problem in the list of valid problems are located using the multimodal large model. The second interface image is output with visual annotations based on the specific location coordinates. Correspondingly, when it is determined that there is a conversion error in the second interface image, outputting the conversion error information includes: If a conversion error is determined to exist in the second interface image, information about the conversion error is output based on the visual annotations.

7. The method according to claim 1, characterized in that, When it is determined that there is a conversion error in the second interface image, the information of the conversion error is output, including: A problem list is generated based on all conversion errors in the second interface image; Determine whether the problem list contains errors that were not recorded in the conversion knowledge base; In cases involving the unrecorded error, a new case record is created based on the unrecorded error; The transformation knowledge base is updated based on the new case records.

8. The method according to claim 1, characterized in that, Before performing several stages of processing on the second interface image using a multimodal large model based on a pre-built transformation knowledge base and the first interface image, the method further includes: Multi-dimensional filtering is performed on the first target image and the second target image. This multi-dimensional filtering includes at least similarity filtering and loading anomaly filtering. The loading anomaly filtering is used to filter the first target image and the second target image when the blank area of ​​either the first target image or the second target image is greater than a first threshold. The similarity filtering is performed through the following steps: The second target image is horizontally mirrored to obtain the mirrored second target image; Calculate the similarity between the mirrored second target image and the first target image; If the similarity is less than the second threshold, filter the first target image and the second target image; Wherein, the first target image is any of the first interface images, and the second target image is the second interface image corresponding to the first target image.

9. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first interface image in a left-to-right layout environment and a second interface image corresponding to the first interface image in a right-to-left layout environment. The processing module is used to perform several stages of processing on the second interface image based on a pre-built conversion knowledge base and the first interface image using a multimodal large model, in order to determine whether there is a conversion error in the second interface image. The several stages of processing include at least a first round of detection and a second round of detection, and the first round of detection and the second round of detection include at least one different detection method. The output module is used to output conversion error information when it is determined that there is a conversion error in the second interface image.

10. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images