Icon identification method of mobile terminal and related device

Abstract

The application discloses a kind of icon identification method, device, storage medium, computer program product and electronic equipment of mobile terminal, comprising: a plurality of initial images including UI icon are collected, positioning mark, random segmentation, classification cutting and edge amplification are carried out to all initial images, and positioning training dataset, first classification training dataset and second classification training dataset are generated;Icon positioning model is trained according to positioning training dataset, first classification model is trained according to first classification training dataset, and second classification model is trained according to second classification training dataset;Real-time capture image to be identified, input the image to be identified into the icon positioning model, obtain the region where target icon is located, and according to the type of target icon region, shunt to first classification model or the second classification model is identified, and obtain identification result;Identification result is input into automatic framework and test operation is executed.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition technology, and more specifically to an icon recognition method, apparatus, storage medium, computer program product, and electronic device for a mobile terminal. Background Technology

[0002] With the rapid development of digitalization, mobile devices have become an indispensable part of life and work. Testing and recognizing UI icons during the mobile device production process is crucial for ensuring the stability and quality of related software. Traditional UI icon recognition methods include manual standard recognition and traditional algorithm comparison recognition. However, both methods are significantly affected by factors such as theme, screen resolution, and background image. For example, the same application may display clearly on high-resolution devices but appear blurry on older, lower-resolution devices, thus affecting the detection results.

[0003] Specifically, traditional template matching methods are poorly adaptable to image variations. Their extracted features are typically designed for specific image conditions and angles. When icons are deformed, rotated, scaled, or subject to changes in lighting, the effectiveness of these features is greatly reduced, leading to a decrease in recognition accuracy. Furthermore, template matching methods rely entirely on pre-prepared icon templates; they cannot effectively recognize icons not included in the template library or those with subtle differences.

[0004] Convolutional neural networks are one of the most widely used deep learning models in the field of image recognition. Through structures such as convolutional layers, pooling layers, and fully connected layers, they automatically learn the feature representation of images, effectively extracting feature information such as texture and shape of icon images, and exhibiting good robustness to changes in icons under different scales, rotation angles, and lighting conditions.

[0005] However, deep learning icon recognition technology based on convolutional neural networks relies heavily on the quality of data annotation. Inaccurate or erroneous annotations may lead to the learning of incorrect features and patterns, resulting in biased recognition results. Furthermore, the size and changing position of the target also affect the final result. Therefore, an improved icon recognition method is needed. Summary of the Invention

[0006] This specification provides an icon recognition method for mobile terminals. By accurately labeling and amplifying image data, and processing the data in layers according to icon characteristics, the method aims to improve recognition accuracy. The method includes:

[0007] Collect several initial images including UI icons, and perform localization annotation, random segmentation, classification cutting and edge augmentation on all initial images to generate localization training dataset, first classification training dataset and second classification training dataset;

[0008] An icon localization model is trained based on the localization training dataset, a first classification model is trained based on the first classification training dataset, and a second classification model is trained based on the second classification training dataset.

[0009] The image to be identified is captured in real time, and the image to be identified is input into the icon localization model to obtain the area where the target icon is located. Based on the type of the area where the target icon is located, the image is sorted into the first classification model or the second classification model for identification to obtain the identification result.

[0010] The recognition results are input into an automated framework to perform test operations.

[0011] Furthermore, the generation of the localization training dataset, the first classification training dataset, and the second classification training dataset includes:

[0012] The UI icons in the initial image are labeled, and the icons are outlined using positioning boxes;

[0013] The initial image is randomly divided into several sub-regions. If the area of ​​the icon in the sub-region is less than 1 / 2 of the positioning box, it is discarded as interference image data; if the area of ​​the icon in the sub-region is greater than or equal to 1 / 2 of the positioning box, it is retained as valid image data.

[0014] Edge augmentation is performed on the icons in the effective image data, and the edge-augmented image data constitutes the localization training dataset;

[0015] The initial image is divided into a status bar area and an application area. If the content of the edge-amplified image data belongs to the status bar area, it constitutes the first classification training dataset; if the content of the edge-amplified image data belongs to the application area, it constitutes the second classification training dataset.

[0016] Preferably, YOLOv6 is used to train the icon positioning model, and MobileNetv2 is used to train the first classification model and the second classification model.

[0017] Furthermore, after inputting the image to be identified into the icon positioning model, it is determined whether the length and width of the image to be identified are less than 256 pixels. If so, edge augmentation is performed on the image to be identified.

[0018] Furthermore, after obtaining the area where the target icon is located, if the area where the target icon is located is the status bar area, the target icon is input into the first classification model for recognition to obtain a first recognition score; if the area where the target icon is located is the application area, the target icon is input into the second classification model for recognition to obtain a second recognition score.

[0019] Furthermore, before inputting the target icon into the first classification model for recognition, it is determined whether the length and width of the target icon are less than 56 pixels. If so, the target icon is augmented at the edges.

[0020] Furthermore, before inputting the target icon into the second classification model for recognition, it is determined whether the length and width of the target icon are less than 100 pixels. If so, the target icon is augmented at the edges.

[0021] Furthermore, after obtaining the first recognition score and / or the second recognition score, the first recognition score is compared with a preset first confidence threshold, and the second recognition score is compared with a preset second confidence threshold. If the first recognition score is higher than the first confidence threshold and / or the second recognition score is higher than the second confidence threshold, then the recognition result is output.

[0022] This specification also provides an icon recognition device for a mobile terminal, comprising:

[0023] The acquisition module is used to acquire several initial images including UI icons, and to perform localization annotation, random segmentation, classification cutting and edge augmentation on all initial images to generate localization training dataset, first classification training dataset and second classification training dataset;

[0024] The model training module is used to train an icon positioning model based on the positioning training dataset, train a first classification model based on the first classification training dataset, and train a second classification model based on the second classification training dataset.

[0025] The real-time recognition module is used to capture the image to be recognized in real time, input the image to be recognized into the icon positioning model to obtain the area where the target icon is located, and according to the type of the area where the target icon is located, it is diverted to the first classification model or the second classification model for recognition to obtain the recognition result;

[0026] An automatic execution module is used to input the recognition results into an automated framework to perform test operations.

[0027] This specification also provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the above-described method.

[0028] This specification also provides a computer program product that stores at least one instruction, which, when executed by a processor, implements the steps of the above-described method.

[0029] This specification also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, the computer program being loaded by the processor and executing the steps of the above-described method.

[0030] This invention effectively improves the accuracy and robustness of icon recognition through an innovative data collection and processing workflow. Compared to traditional template matching techniques, it is no longer limited to feature extraction under specific image conditions and angles, and can flexibly handle complex situations such as icon deformation, rotation, scaling, and lighting changes. It can accurately grasp the essential features of icons and avoid subtle differences interfering with recognition. The precise annotation and edge augmentation strategies in data processing provide high-quality data support for model learning and reduce misjudgments. Furthermore, this invention utilizes a multi-model collaborative architecture, processing icons in layers according to their characteristics, to further enhance recognition accuracy and efficiency. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating an icon recognition method for a mobile terminal provided in an embodiment of this specification.

[0032] Figure 2 This is a schematic diagram of the structure of an icon recognition device for a mobile terminal provided in an embodiment of this specification.

[0033] Figure 3 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0035] The icon recognition method provided in this specification is applied to a mobile terminal, which may be a smartphone, tablet computer, smartwatch, or other device, but is not limited to these.

[0036] Please see Figure 1This is a flowchart illustrating a method for icon recognition on a mobile terminal, provided in an embodiment of this specification. Specifically, the icon recognition method for the mobile terminal in this embodiment may include the following steps:

[0037] S1: Collect several initial images including UI icons, and perform localization annotation, random segmentation, classification cutting and edge augmentation on all initial images to generate localization training dataset, first classification training dataset and second classification training dataset.

[0038] In one or more embodiments of this specification, image acquisition can be performed using automated software screenshotting. That is, through specific software instructions or operations, all or part of the content currently displayed on the mobile terminal device's screen is saved as an image file. In icon recognition, screenshots can be used as a way to obtain icon images, especially when recognizing application icons that are installed on the mobile terminal device and are running; the screen area including the icon can be directly captured as input data for recognition. It should be understood that, in addition to screenshots, images including UI icons can also be obtained through other methods.

[0039] In one or more embodiments of this specification, the collected images may include screenshots of UI interfaces from different brands of terminal devices, different operating systems / versions, or different application scenarios, in order to construct diverse training datasets and improve the generalization ability of the model.

[0040] In one or more embodiments of this specification, generating the localization training dataset, the first classification training dataset, and the second classification training dataset may specifically include the following steps:

[0041] Step S11: Annotate the UI icons in the initial image by framing the icons with positioning boxes. In one or more embodiments of this specification, annotation tools such as Labellmg, CVAT, etc., can be used for annotation.

[0042] Step S12: The initial image is randomly divided into several sub-regions. If the area of ​​the icon in a sub-region is less than 1 / 2 of the positioning box, it is discarded as interference image data; if the area of ​​the icon in a sub-region is greater than or equal to 1 / 2 of the positioning box, it is retained as valid image data.

[0043] Because the UI icons are small, they need to be augmented to improve training accuracy. Therefore, the following steps are performed:

[0044] Step S13 involves edge augmentation of the icons in the valid image data. The edge-augmented image data constitutes the localization training dataset. More specifically, edge augmentation includes expanding the icon's side length outward by 1 / 4. For example, if the original icon is 40×40 pixels, it will be expanded to 50×50 pixels. It should be understood that the edge augmentation ratio can be adjusted according to the icon size. For example, for small icons with a side length of less than 20 pixels, an additional 10% augmentation ratio can be added to compensate for feature loss.

[0045] Step S14: Divide the initial image into a status bar area and an application area. If the content of the image data after edge augmentation belongs to the status bar area, it constitutes the first classification training dataset; if the content of the image data after edge augmentation belongs to the application area, it constitutes the second classification training dataset.

[0046] S2, train the icon localization model based on the localization training dataset, train the first classification model based on the first classification training dataset, and train the second classification model based on the second classification training dataset.

[0047] In one or more embodiments of this specification, YOLOv6 or later versions are selected for training the icon localization model. That is, the localization training dataset is input into the YOLOv6 or later version model for training to obtain the icon localization model.

[0048] Since the classification models run on mobile devices and need to be adapted to low-end models, MobileNetV2 is chosen for training the first and second classification models. Specifically, the first classification training dataset is input into the MobileNetV2 model for training to obtain the first classification model; the second classification training dataset is input into the MobileNetV2 model for training to obtain the second classification model.

[0049] S3 captures the image to be recognized in real time, inputs the image to be recognized into the icon localization model, obtains the area where the target icon is located, and, according to the type of the area where the target icon is located, splits it into the first classification model or the second classification model for recognition, and obtains the recognition result.

[0050] In one or more embodiments of this specification, the image to be identified may be a screenshot of a UI interface captured by automated software, or a partial image of a region of interest (ROI).

[0051] In one or more embodiments of this specification, after the image to be recognized is input into the icon positioning model, it is determined whether the length and width of the image to be recognized are less than 256 pixels. If so, edge augmentation is performed. More specifically, color augmentation is performed using a color that is the same as or similar to the edge of the image to be recognized, so that the length and width of the image to be recognized reach 256 pixels.

[0052] Icon localization models can quickly locate the approximate area where a target icon is located. Built upon diverse training datasets, these models efficiently filter out a large amount of irrelevant pixel information in complex backgrounds and varied UI layouts. Even on high-resolution large-screen phones or interfaces with dense icon arrangements, they can quickly pinpoint the local area where the target icon is located, effectively mitigating localization errors caused by minor icon displacement.

[0053] After obtaining the area where the target icon is located, if the area where the target icon is located is the status bar area, the target icon is input into the first classification model for recognition to obtain the first recognition score; if the area where the target icon is located is the application area, the target icon is input into the second classification model for recognition to obtain the second recognition score.

[0054] Before inputting the target icon into the first classification model for recognition, it is determined whether the length and width of the target icon are less than 56 pixels. If so, edge augmentation is performed. More specifically, color augmentation is performed using the same or similar color as the edge of the target icon to make the length and width of the target icon reach 56 pixels.

[0055] Before inputting the target icon into the second classification model for recognition, it is determined whether the length and width of the target icon are less than 100 pixels. If so, edge augmentation is performed. Similarly, color augmentation is performed using the same or similar color as the edge of the target icon to make the length and width of the target icon reach 100 pixels.

[0056] After obtaining the first recognition score and / or the second recognition score, the first recognition score is compared with a preset first confidence threshold, and the second recognition score is compared with a preset second confidence threshold. If the first recognition score is higher than the first confidence threshold and / or the second recognition score is higher than the second confidence threshold, it is output as the recognition result; otherwise, it is judged as a recognition error, that is, the target icon does not exist in the initial UI icon data.

[0057] S4 inputs the recognition results into the automation framework to perform test operations. The automation framework uses the results to perform operations such as matching, clicking, moving, and confirming existence, and completes the automation script to confirm the production quality of the mobile terminal UI software. It ensures feedback in cases where the mobile terminal UI icons are different or some icons may disappear, and avoids quality problems caused by the software system during the production process, which may affect the final product quality.

[0058] Please see Figure 2 This is a schematic diagram illustrating the structure of an icon recognition device for a mobile terminal provided in an embodiment of this specification. This icon recognition device for the mobile terminal can be implemented as all or part of an electronic device through software, hardware, or a combination of both. For example... Figure 2As shown, in one or more embodiments of this specification, the icon recognition device 1 of the mobile terminal includes a model data acquisition module 11, a model training module 12, a real-time recognition module 13, and an automatic execution module 14, specifically including:

[0059] The acquisition module 11 is used to acquire several initial images including UI icons, and to perform localization annotation, random segmentation, classification cutting and edge augmentation on all initial images to generate a localization training dataset, a first classification training dataset and a second classification training dataset.

[0060] The model training module 12 is used to train an icon localization model based on the localization training dataset, train a first classification model based on the first classification training dataset, and train a second classification model based on the second classification training dataset.

[0061] The real-time recognition module 13 is used to capture the image to be recognized in real time, input the image to be recognized into the icon positioning model, obtain the area where the target icon is located, and according to the type of the area where the target icon is located, it is diverted to the first classification model or the second classification model for recognition to obtain the recognition result.

[0062] The automatic execution module 14 is used to input the recognition results into the automation framework to perform test operations.

[0063] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0064] This specification also provides an embodiment of a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figure 1 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figure 1 The specific details of the illustrated embodiments will not be elaborated here.

[0065] This specification also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor as described above. Figure 1 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figure 1 The specific details of the illustrated embodiments will not be elaborated here.

[0066] The embodiments in this specification also provide Figure 3 The diagram shows the structure of the electronic device. Figure 3At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile storage, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile storage into memory and then runs it to implement the aforementioned network video skinning method.

[0067] The solutions described in the embodiments of this specification can also be applied to the field of icon recognition for smart home devices' interactive interfaces. Smart home control panels contain diverse application icons that vary depending on the device brand, model, and functional expansion. By introducing the above solutions, automated screenshots can be used to collect device interactive interface icons, build an adaptive dataset to train a model, accurately identify the icons corresponding to control commands, and improve the smart home interactive experience.

[0068] The above-described solutions in the embodiments of this specification can also be applied to intelligent vehicle central control systems. As the demand for in-vehicle application icon recognition increases with the integration of functions, the above solutions can optimize the icon recognition module. By collecting icon data under different driving scenarios and in-vehicle system versions to train the model, the operation range and function of the icon can be accurately determined, thereby enhancing driving safety and operational convenience and helping to upgrade the intelligent interaction of intelligent vehicles.

[0069] Meanwhile, we can explore replacing the existing model with a more advanced neural network architecture to improve learning efficiency; optimize data annotation technology to reduce manual annotation errors and improve annotation speed through intelligent algorithms; and introduce reinforcement learning mechanisms to dynamically optimize the model based on user operation feedback, so as to achieve a continuous leap in icon recognition performance.

[0070] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0071] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0072] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0073] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0074] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0075] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0080] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0081] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0082] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0085] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0086] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for icon recognition on a mobile terminal, characterized in that, include: Collect several initial images including UI icons, and perform localization annotation, random segmentation, classification cutting and edge augmentation on all initial images to generate localization training dataset, first classification training dataset and second classification training dataset; An icon localization model is trained based on the localization training dataset, a first classification model is trained based on the first classification training dataset, and a second classification model is trained based on the second classification training dataset. The image to be identified is captured in real time, and the image to be identified is input into the icon localization model to obtain the area where the target icon is located. Based on the type of the area where the target icon is located, the image is sorted into the first classification model or the second classification model for identification to obtain the identification result. The recognition results are input into an automated framework to perform test operations.

2. The icon recognition method for a mobile terminal according to claim 1, characterized in that, The generated localization training dataset, the first classification training dataset, and the second classification training dataset include: The UI icons in the initial image are labeled, and the icons are outlined using positioning boxes; The initial image is randomly divided into several sub-regions. If the area of ​​the icon in the sub-region is less than 1 / 2 of the positioning box, it is discarded as interference image data; if the area of ​​the icon in the sub-region is greater than or equal to 1 / 2 of the positioning box, it is retained as valid image data. Edge augmentation is performed on the icons in the effective image data, and the edge-augmented image data constitutes the localization training dataset; The initial image is divided into a status bar area and an application area. If the content of the edge-amplified image data belongs to the status bar area, it constitutes the first classification training dataset; if the content of the edge-amplified image data belongs to the application area, it constitutes the second classification training dataset.

3. The icon recognition method for a mobile terminal according to claim 1, characterized in that, YOLOv6 was selected for training the icon localization model, and MobileNetv2 was selected for training the first classification model and the second classification model.

4. The icon recognition method for a mobile terminal according to claim 1, characterized in that, After the image to be identified is input into the icon positioning model, it is determined whether the length and width of the image to be identified are less than 256 pixels. If so, edge augmentation is performed on the image to be identified.

5. The icon recognition method for a mobile terminal according to claim 1, characterized in that, After obtaining the area where the target icon is located, if the area where the target icon is located is the status bar area, the target icon is input into the first classification model for recognition to obtain a first recognition score; if the area where the target icon is located is the application area, the target icon is input into the second classification model for recognition to obtain a second recognition score.

6. The icon recognition method for a mobile terminal according to claim 5, characterized in that, Before inputting the target icon into the first classification model for recognition, it is determined whether the length and width of the target icon are less than 56 pixels. If so, the target icon is augmented at the edges.

7. The icon recognition method for a mobile terminal according to claim 5, characterized in that, Before inputting the target icon into the second classification model for recognition, it is determined whether the length and width of the target icon are less than 100 pixels. If so, the target icon is augmented at the edges.

8. The icon recognition method for a mobile terminal according to claim 5, characterized in that, After obtaining the first recognition score and / or the second recognition score, the first recognition score is compared with a preset first confidence threshold, and the second recognition score is compared with a preset second confidence threshold. If the first recognition score is higher than the first confidence threshold and / or the second recognition score is higher than the second confidence threshold, then the recognition result is output.

9. An icon recognition device for a mobile terminal, characterized in that, include: The data acquisition module is used to acquire several initial images including UI icons, and to perform localization annotation, random segmentation, classification cutting and edge augmentation on all initial images to generate localization training dataset, first classification training dataset and second classification training dataset; The model training module is used to train an icon positioning model based on the positioning training dataset, train a first classification model based on the first classification training dataset, and train a second classification model based on the second classification training dataset. The real-time recognition module is used to capture the image to be recognized in real time, input the image to be recognized into the icon positioning model to obtain the area where the target icon is located, and according to the type of the area where the target icon is located, it is diverted to the first classification model or the second classification model for recognition to obtain the recognition result; An automatic execution module is used to input the recognition results into an automated framework to perform test operations.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer program product having at least one instruction stored thereon, characterized in that, When the at least one instruction is executed by the processor, it implements the steps of the method according to any one of claims 1 to 8.

12. An electronic device, comprising: A processor and a memory, characterized in that the memory stores a computer program, which is loaded by the processor and executes the steps of the method as claimed in any one of claims 1 to 8.

Images