Method and device for generating training sample, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a method and device for generating training samples, electronic equipment, a computer readable storage medium and a computer program product; the method comprises: obtaining a plurality of canvas elements in a page, the plurality of canvas elements each having a corresponding priority, each canvas element comprising a plurality of components, screening the plurality of canvas elements based on the priority of each canvas element to obtain a target canvas element used as a top canvas of the page, determining the hierarchical relationship between the target canvas element and the plurality of components in the target canvas element, screening the plurality of components based on the hierarchical relationship to obtain target components that can be displayed in the page, determining the position of the target components in the target canvas element, and constructing a sample label based on the position, labeling a page image corresponding to the page, and generating training samples for training a component recognition model. Through the application, the time required to generate training samples can be reduced, and the efficiency of generating training samples can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating training samples. Background Technology

[0002] In related technologies, in order to achieve component recognition through models, training samples can be obtained by manually annotating the controls on the page. However, obtaining training samples by manual annotation takes a long time, which means that the efficiency of generating training samples in related technologies is low. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating training samples, which can reduce the time required to generate training samples and improve the efficiency of generating training samples.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a method for generating training samples, the method comprising:

[0006] Get multiple canvas elements in the page, each of which has a corresponding priority, and each canvas element includes multiple components;

[0007] Based on the priority of each canvas element, the multiple canvas elements are filtered to obtain the target canvas element to be used as the top-level canvas of the page.

[0008] Determine the target canvas element and the hierarchical relationship between multiple components within the target canvas element;

[0009] Based on the hierarchical relationship, the multiple components are filtered to obtain the target components that can be displayed on the page;

[0010] The position of the target component within the target canvas element is determined, and sample labels are constructed based on the position. The page image corresponding to the page is then labeled to generate training samples for training the component recognition model.

[0011] This application provides a training sample generation apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire multiple canvas elements in the page. Each of the multiple canvas elements has a corresponding priority, and each canvas element includes multiple components.

[0013] A filtering module is used to filter the multiple canvas elements based on the priority of each canvas element to obtain the target canvas element to be used as the top-level canvas of the page.

[0014] A determination module is used to determine the target canvas element and the hierarchical relationship between multiple components in the target canvas element;

[0015] The filtering module is also used to filter the multiple components based on the hierarchical relationship to obtain target components that can be displayed on the page;

[0016] The determining module is further configured to determine the position of the target component in the target canvas element, construct sample labels based on the position, annotate the page image corresponding to the page, and generate training samples for training the component recognition model.

[0017] In the above scheme, the priority of the canvas elements includes inter-layer priority for controlling the rendering order. The filtering module is also used to sort the multiple canvas elements in descending order of the inter-layer priority to obtain a canvas element sequence; obtain the area of ​​the page, and filter the multiple canvas elements based on the area of ​​the page and the canvas element sequence to obtain the target canvas element used as the top-level canvas of the page.

[0018] In the above scheme, the filtering module is further configured to determine the first area ratio of the canvas elements according to the order of the canvas elements in the canvas element sequence; wherein, the first area ratio is the ratio of the area of ​​the canvas element to the area of ​​the page; and the first canvas element in the canvas element sequence whose first area ratio is greater than the first ratio threshold is determined as the target canvas element.

[0019] In the above scheme, the filtering module is further configured to determine, based on the hierarchical relationship, components with a hierarchy level lower than a hierarchy threshold as first components, wherein there are multiple first components; and to filter the multiple first components based on the hierarchy level and area of ​​the first components to obtain target components that can be displayed on the page.

[0020] In the above scheme, the multiple first components include multiple levels. For any one first component, the first component at the upper level of the first component is the parent component of the first component, and the first component at the lower level of the first component is the child component of the first component. The filtering module is further configured to, for each first component in the level, determine the first component and its corresponding child component as target components that can be displayed on the page when the first component meets the target conditions. The target conditions are: when the first component does not have a parent component, the second area ratio of the first component is greater than or equal to the second ratio threshold, where the second area ratio is the ratio of the area of ​​the first component to the area of ​​the target canvas element; when the first component has a parent component, the third area ratio of the parent component of the first component is greater than or equal to the second ratio threshold, where the third area ratio is the ratio of the area of ​​the parent component of the first component to the area of ​​the target canvas element; when the first component has a parent component, the third area ratio of the parent component of the first component is less than the second ratio threshold, and the second area ratio of the first component is greater than or equal to the second ratio threshold.

[0021] In the above scheme, the multiple first components include a first level and a second level. The filtering module is further configured to determine the first component and its sub-components as target components that can be displayed on the page if the second area ratio of the first component in the first level is greater than or equal to the second ratio threshold. The second area ratio is the area ratio of the first component to the target canvas element, and the second level of the sub-component is the next level after the first level.

[0022] If the second area ratio of the first component in the first level is less than the second ratio threshold, and the second area ratio of the first component in the second level is greater than or equal to the second ratio threshold, then the first component in the second level is determined as a target component that can be displayed on the page.

[0023] In the above scheme, the filtering module is further used to determine the transparency of the image component after identifying the component with a level lower than the level threshold among the multiple components as the first component, when the first component is an image component.

[0024] The following conditions are met to trigger the process of filtering multiple first components based on the hierarchy and area of ​​the first component to obtain a target component that can be displayed on the page: the transparency of the image component is greater than a transparency threshold; or the area of ​​the image component is less than a third area threshold.

[0025] In the above scheme, there are multiple target components, including a second component for displaying text. The determining module is further configured to, after determining the position of the target component in the target canvas element, if the target component includes a third component, determine the collision relationship between the second component and the third component on the page based on the positions of the second component and the third component; wherein the distance between the third component and the second component is less than a third distance threshold; if the collision relationship indicates that there is a collision between the second component and the third component, determine the target position based on the positions of the second component and the third component; construct sample labels based on the target positions, and annotate the page image corresponding to the page to obtain training samples for training the collision recognition model.

[0026] In the above scheme, the second component and the third component are both rectangular in shape. The position of the second component includes a first angular coordinate and a second angular coordinate, and the position of the third component includes a third angular coordinate and a fourth angular coordinate. The first angular coordinate, the second angular coordinate, the third angular coordinate, and the fourth angular coordinate are in the same two-dimensional coordinate system. The first angular coordinate and the third angular coordinate correspond to each other in the rectangle, and the second angular coordinate and the fourth angular coordinate correspond to each other in the rectangle.

[0027] The determining module is further configured to determine that the collision relationship is that the second component and the third component collide on the page if the x-coordinate of the first angular coordinate is greater than the x-coordinate of the third angular coordinate and the x-coordinate of the second angular coordinate is less than the x-coordinate of the fourth angular coordinate.

[0028] In the above scheme, the determining module is also used to determine the difference between the ordinate of the second angular coordinate and the ordinate of the fourth angular coordinate;

[0029] If the difference is greater than or equal to the difference threshold, the second angular coordinate and the fourth angular coordinate are determined as the coordinates of the target position;

[0030] If the difference is less than the difference threshold, the ordinate of the first angular coordinate and the ordinate of the third angular coordinate are determined as the ordinate of the target position, and the abscissa of the second angular coordinate and the abscissa of the fourth angular coordinate are determined as the abscissa of the target position.

[0031] In the above scheme, the determining module is further used to obtain the world coordinates of the target component in the world coordinate system; transform the world coordinates from the world coordinate system to the view coordinate system to obtain the view coordinates; and map the view coordinates to the display area in the page to obtain the position of the target component in the target canvas element.

[0032] In the above scheme, the priority includes inter-layer priority and intra-layer priority used to control the rendering order; the filtering module is also used to determine the candidate canvas element with the highest inter-layer priority from the multiple canvas elements and count the number of the candidate canvas elements; if the number is greater than 1, the candidate canvas element with the highest intra-layer priority is determined as the target canvas element.

[0033] In the above scheme, the filtering module is further used to determine the canvas element with the highest priority among the plurality of canvas elements as the first canvas element; based on the first canvas element and the priority of the canvas elements, filter the other canvas elements among the plurality of canvas elements to obtain the second canvas element; and determine the first canvas element and the second canvas element as the target canvas elements to be used as the top-level canvas of the page.

[0034] In the above scheme, the filtering module is further configured to, for the other canvas elements, if there are other canvas elements with a first overlapping area smaller than a first area threshold, and the distance between the first canvas element and the other canvas elements is less than or equal to a first distance threshold, then determine the other canvas element with the highest priority as the second canvas element; wherein, the first overlapping area is the overlapping area between the other canvas element and the first canvas element on the page; if there are other canvas elements with a second overlapping area smaller than the first area threshold, and the distance between the other canvas element and the target area is less than or equal to a second distance threshold, then determine the other canvas element with the highest priority as the second canvas element; wherein, the second overlapping area is the overlapping area between the other canvas element and the target area on the page, and the target area is the sum of the area of ​​the first canvas element on the page and the area of ​​the second canvas element on the page.

[0035] In the above scheme, the hierarchical relationship is represented by a tree structure including multiple nodes, wherein the tree structure has the target canvas element as the root node and the components in the target canvas element as non-root nodes; the filtering module is further configured to determine the component corresponding to the child node of the root node as the target component that can be displayed on the page if the number of the target canvas elements is 1; if the number of the target canvas elements is multiple and the overlap area between any two target canvas elements is less than a second area threshold, determine the component corresponding to the child node of the root node as the target component that can be displayed on the page.

[0036] This application provides an electronic device, the electronic device comprising:

[0037] Memory is used to store executable instructions or computer programs.

[0038] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training sample generation method provided in the embodiments of this application.

[0039] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the training sample generation method provided in this application when executed by a processor.

[0040] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the training sample generation method provided in this application.

[0041] The embodiments of this application have the following beneficial effects:

[0042] The training sample generation method provided in this application embodiment can obtain multiple canvas elements on a page, each with a corresponding priority. Each canvas element includes multiple components. Based on the priority of each canvas element, the multiple canvas elements are filtered to obtain the target canvas element used as the top-level canvas of the page. This is equivalent to performing a first filtering based on the priority of the canvas elements to obtain the target canvas element. The hierarchical relationship between the target canvas element and its multiple components is determined. Based on the hierarchical relationship, the multiple components are filtered to obtain the target components that can be displayed on the page. This is equivalent to performing a second filtering based on the hierarchical relationship to obtain the target components. Then, the position of the target component in the target canvas element can be determined, and sample labels are constructed based on the position to annotate the corresponding page image, generating training samples for training the component recognition model.

[0043] As can be seen, in the training sample generation method provided in this application embodiment, the canvas elements can be first filtered according to their priority to obtain target canvas elements. Then, based on the hierarchical relationship, multiple components in the target canvas elements are filtered a second time to obtain target components. In other words, for the canvas elements on the page, two different filters can be performed to obtain accurate target components that can be displayed on the page, without requiring the user to manually mark the components on the page. This reduces the time required to determine the target components on the page and improves the efficiency of determining the target components on the page. Furthermore, sample labels can be constructed based on the position of the target components to generate training samples, thereby reducing the time required to generate training samples, improving the efficiency of generating training samples, and improving the accuracy of generating training samples. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of the training sample generation system provided in the embodiments of this application;

[0045] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0046] Figure 3 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 1 ;

[0047] Figure 4 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 2 ;

[0048] Figure 5 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 3 ;

[0049] Figure 6 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 4 ;

[0050] Figure 7 This is a schematic diagram of a page provided in an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of the hierarchical relationship provided in the embodiments of this application;

[0052] Figure 9 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 5 ;

[0053] Figure 10 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 6 ;

[0054] Figure 11 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 7 ;

[0055] Figure 12 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 8 ;

[0056] Figure 13 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 9 ;

[0057] Figure 14 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 10 ;

[0058] Figure 15 This is a schematic diagram of the collision relationship provided in the embodiments of this application;

[0059] Figure 16 This is a schematic diagram of the target area provided in the embodiments of this application;

[0060] Figure 17 This is a schematic diagram of the component identification model provided in the embodiments of this application;

[0061] Figure 18 This is a schematic diagram of the output of the component identification model provided in the embodiments of this application. Figure 1 ;

[0062] Figure 19 This is a schematic diagram of the collision recognition model provided in the embodiments of this application;

[0063] Figure 20 This is a schematic diagram of the output of the collision recognition model provided in the embodiments of this application. Figure 1 ;

[0064] Figure 21 This is a schematic diagram of the output of the collision recognition model provided in the embodiments of this application. Figure 2 ;

[0065] Figure 22 This is a schematic diagram of the output of the collision recognition model provided in the embodiments of this application. Figure 3 ;

[0066] Figure 23 This is a schematic diagram of the output of the component identification model provided in the embodiments of this application. Figure 2 ;

[0067] Figure 24 This is a schematic diagram of the output of the component identification model provided in the embodiments of this application. Figure 3 . Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0070] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0071] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0072] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0073] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0074] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0075] 1) Page: A page refers to a collection of information or content presented on the display screen of an electronic device. A page is a complete visual unit displayed in an electronic device. Users can browse different pages by clicking, scrolling, turning pages, etc. In the training sample generation method provided in the embodiments of this application, the page can be a game page, an instant messaging page, an online education page, an e-shopping page, an online office page, a multimedia playback page, etc., which can be determined according to actual usage needs.

[0076] 2) The Canvas element, also known as a canvas component, is a visual container used to hold components (also called elements; interactive components can be called controls) within the canvas. Components can be various, including text components, image components, button components, checkbox controls, slider components, scrollbar components, dropdown lists, text box components, scroll view components, mask components, and special effects components.

[0077] 3) Training samples, also known as training sets, are data sets used to train machine learning models or artificial intelligence algorithms. Training samples can include input data and corresponding output labels. By learning from training samples, the model can learn the relationship between inputs and outputs, thereby predicting or classifying new input data. Here, new input data is only used to distinguish it from the original input data.

[0078] During the research process, the inventors discovered the following technical problems in the relevant technology:

[0079] 1. In related technologies, in order to achieve component recognition through models, training samples can be obtained by manually annotating the controls on the page. However, obtaining training samples by manual annotation takes a long time, which means that the efficiency of generating training samples in related technologies is low.

[0080] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating training samples, which can reduce the time required to generate training samples and improve the efficiency of generating training samples.

[0081] See Figure 1 , Figure 1 This is a schematic diagram of the structure of the training sample generation system provided in the embodiment of this application. The training sample generation system 100 is designed to support the generation of a training sample. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of both.

[0082] In some embodiments, when a user wants to generate training samples, in response to a training sample generation instruction triggered for terminal 400, terminal 400 can send a training sample generation request to server 200. After receiving the training sample generation request, server 200 can obtain multiple canvas elements on the page. The multiple canvas elements have corresponding priorities, and each canvas element includes multiple components.

[0083] After obtaining multiple canvas elements on the page, the server 200 can filter the multiple canvas elements based on the priority of each canvas element to obtain the target canvas element to be used as the top-level canvas of the page. It can determine the hierarchical relationship between the target canvas element and multiple components in the target canvas element, filter the multiple components based on the hierarchical relationship to obtain the target components that can be displayed on the page, determine the position of the target components in the target canvas element, construct sample labels based on the position, label the page image corresponding to the page, and generate training samples for training the component recognition model.

[0084] After generating training samples, server 200 can return the generation result, which represents the completion of training sample generation for training the component recognition model, to terminal 400. Terminal 400 can then display the generation result on graphical interface 410-1 so that the user can know that the training samples for training the component recognition model have been generated.

[0085] In the training sample generation method provided in this application embodiment, canvas elements can be first filtered according to their priority to obtain target canvas elements. Then, based on the hierarchical relationship, multiple components in the target canvas elements are filtered a second time to obtain target components. In other words, for canvas elements on the page, two different filters can be performed to obtain accurate target components that can be displayed on the page, without requiring the user to manually mark the components on the page. This reduces the time required to determine the target components on the page and improves the efficiency of determining the target components. Furthermore, sample labels can be constructed based on the position of the target components to generate training samples, thereby reducing the time required to generate training samples, improving the efficiency of generating training samples, and improving the accuracy of generating training samples.

[0086] The training sample generation method provided in this application can be applied to multiple fields. Specifically, in some embodiments, this method can be applied to the gaming field. In the gaming field, the page can be any display page corresponding to the game, and the target component can be a game component on the game's display page that triggers corresponding instructions. For example, the game's display page can be the game's homepage, and the component can be a start component for starting the game. In this case, the training sample is the training sample corresponding to the game page.

[0087] In some embodiments, the training sample generation method provided in this application can be applied to the field of instant messaging. In the context of instant messaging, the page can be any display page corresponding to instant messaging, and the target component can be an instant messaging component on the display page that triggers corresponding instructions. For example, the instant messaging display page can be a conversation interface for a first and second object to have a conversation, and the component can be a call component for initiating a video call. In this case, the training sample is the training sample corresponding to the instant messaging page.

[0088] In some embodiments, the training sample generation method provided in this application can be applied to the field of online education. In the context of online education, the page can be any display page corresponding to the online education content, and the target component can be a control component on the online education display page that triggers corresponding instructions. For example, the online education display page can be the live streaming interface of a live classroom, and the component can be a sending component for sending bullet comments. In this case, the training sample is the training sample corresponding to the online education page.

[0089] In some embodiments, the training sample generation method provided in this application can be applied to the field of e-commerce. In the context of e-commerce, the page can be any display page corresponding to e-commerce, and the target component can be a control component on the e-commerce display page that triggers corresponding instructions. For example, the e-commerce display page can be a product display interface, and the component can be a scroll bar component for scrolling and displaying products. In this case, the training sample is the training sample corresponding to the e-commerce page.

[0090] In some embodiments, the training sample generation method provided in this application can be applied to the field of online office work. For the online office field, the page can be any display page corresponding to online office work, and correspondingly, the target component can be a control component in the online office display page that triggers corresponding instructions. For example, the online office display page can be the display interface of an online meeting, and the component can be a screen sharing component for sharing the display screen. In this case, the training sample is the training sample corresponding to the online office page.

[0091] In some embodiments, the training sample generation method provided in this application can be applied to the field of multimedia playback. In the context of multimedia playback, the page can be any display page corresponding to multimedia playback, and the target component can be a control component on the display page of multimedia playback that triggers corresponding instructions. For example, the display page of multimedia playback can be a video display interface, and the component can be a pause control for pausing a playing video. In this case, the training sample is the training sample corresponding to the multimedia playback page.

[0092] The following describes the electronic device provided in the embodiments of this application. The electronic device implementing the training sample generation method of the embodiments of this application can be a terminal, a server, or a combination of both. The terminal can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals.

[0093] In some embodiments, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and server can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0094] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0095] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0096] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0097] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0098] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0099] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0100] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0101] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0102] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0103] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0104] In some embodiments, the training sample generation apparatus provided in this application can be implemented in software. Figure 2 A training sample generation device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 4551, a filtering module 4552, and a determination module 4553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0105] In other embodiments, the training sample generation apparatus provided in this application can be implemented in hardware. As an example, the training sample generation apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training sample generation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0106] In some embodiments, the terminal or server can implement the training sample generation method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as applications for training models or applications for generating training samples; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0107] The method for generating training samples provided in the embodiments of this application will be described below. As mentioned above, the electronic device implementing the method for generating training samples in the embodiments of this application can be a terminal, a server, or a combination of both. See also Figure 3, Figure 3 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 1 The following is combined with Figure 3 The steps shown are illustrated using an electronic device that is a server as an example to explain the method for generating training samples provided in the embodiments of this application.

[0108] In step 101, multiple canvas elements on the page are obtained.

[0109] In practical applications, when a user wants to generate training samples, in response to the user's command to generate training samples triggered by the terminal, the terminal can send a training sample generation request to the server. After receiving the training sample generation request, the server 200 can obtain multiple canvas elements of the page.

[0110] The following describes the pages provided in the embodiments of this application. The page is the page corresponding to the scene. The page corresponding to the scene refers to the page of the application displayed on the electronic device. There can be one or more pages. For example, for a game application, the scene can include any one of the following: main menu scene, game level scene, settings menu scene, and loading scene.

[0111] A scene can have one or more pages. For example, a loading scene can have only one page: the loading page. Similarly, a game level scene can have multiple pages, which may include at least two of the following: a page displaying the virtual scene, a page displaying information about virtual objects, and a page displaying scores. This is just an example; the specific settings can be adjusted based on actual usage.

[0112] The following explains the relationship between canvas elements and pages. In some embodiments, a page for a scene corresponds to a canvas element, and all components on the page belong to the canvas element. This approach enables unified management of components and can be applied to relatively simple user interfaces (UI).

[0113] In some embodiments, a scene's page corresponds to multiple canvas elements. When there are multiple pages in a scene, each page corresponds to one or more canvas elements. For pages with multiple canvas elements, the page can contain multiple regions, effectively managing the corresponding pages through multiple canvas element partitions. The number of regions can be greater than, less than, or equal to the number of canvas elements. One canvas can correspond to one or more regions, and one region can contain one or more canvases.

[0114] In some embodiments, when a page corresponds to multiple canvas elements, there is a hierarchical relationship between the multiple canvas elements. This is equivalent to multiple canvas elements managing the corresponding page in layers, and one layer can correspond to one or more canvas elements.

[0115] The following explains the difference between pages with multiple canvas element partition management and pages with multiple canvas element layer management. Pages with multiple canvas element partition management refer to pages that can be divided into multiple functional areas, each managed by one or more canvas elements. These functional areas can be side-by-side, and there should be no overlap between them. The overlap area between canvas elements corresponding to different functional areas should be less than a certain threshold. For example, a page can have three functional areas: functional area 1 corresponds to the navigation bar, functional area 2 corresponds to the content display area, and functional area 3 corresponds to the sidebar.

[0116] Layered management of multiple canvas elements refers to dividing the canvas elements of a page into multiple layers. Each layer contains one or more canvas elements, and the canvas elements of different layers overlap. The canvas elements of different layers are responsible for managing a specific visual effect or a specific function. For example, a page may have multiple layers: layer 1 is the background layer, layer 2 may be the content display layer, and layer 3 may be the pop-up layer (corresponding to pop-ups or floating layers).

[0117] In some embodiments, multiple rendering modes exist for a page, including a first rendering mode (Screen Space-Overlay), a second rendering mode (Screen Space-Camera), and a third rendering mode (World Space). In the first rendering mode, canvas elements and components are always rendered on top of the screen and are not controlled by any camera. In the second rendering mode, canvas elements and components are associated with the corresponding camera and are rendered according to the camera's perspective. In the third rendering mode, canvas elements and components can be treated as three-dimensional objects in the scene corresponding to the page, and can be rotated, scaled, and so on.

[0118] Each canvas element contains multiple components; the canvas element is the container used to hold these components. A canvas element can include at least one component, which can include text components, image components, button components, checkbox controls, slider components, scrollbar components, dropdown lists, text box components, scroll view components, mask components, and special effects components.

[0119] Text components can be used to display pre-defined text. In other words, text components are used to display non-interactive text content within a canvas element. The content, font, and color of the text can be preset for text components. For example, in the gaming field, text components can be used to display one or more of the following: the score obtained by a virtual object in the game interface, the virtual object's health points, and the virtual object's status information.

[0120] Image components can be used to display pre-defined images. In other words, image components are used to display non-interactive images within canvas elements. The images to be displayed and their display styles can be pre-defined for image components. For example, image components can be used as background images for buttons, icons, or avatars, depending on the specific usage requirements.

[0121] Button components are used to implement interaction. For button components, you can set the text displayed on the button, the image displayed on the button, the trigger command method, the corresponding event of the command, etc. The trigger command method can include one or more of the following: single click, double click, and press. The corresponding event is the event generated after the button component triggers the command. The event can be set according to the actual usage requirements. The event can be to start a function, jump to a page, adjust parameters, etc.

[0122] Checkbox controls can be used to create toggleable checkboxes. They allow you to configure whether a checkbox is selected, what events are triggered when a checkbox is selected, and what events are triggered when a checkbox is not selected. The number of checkboxes can be set according to actual usage needs. When there are two checkboxes, the checkbox control can also be called a switch component, used to determine whether the switch is currently on or off.

[0123] When there are multiple checkboxes, they can be combined into a switch group control. The switch group control can be used to integrate a group of mutually exclusive switch controls. Switch controls belonging to the same group will be constrained, that is, only one switch component can be selected at a time, and selecting one switch component will automatically deselect all other switch components.

[0124] The slider component allows users to select a value by sliding it. For example, the slider component can display a decimal value, and users can drag it between the minimum and maximum values. Within the canvas element, the slider can be horizontal or vertical, and can also be set to any direction depending on the actual usage requirements.

[0125] A scrollbar component allows users to view content by scrolling. For example, a scrollbar component may have a decimal value between 0 and 1. This value will change as the user drags the scrollbar.

[0126] The dropdown menu displays a list of available options, which can include multiple candidates. These candidates can be any of the following: a specified text string or an image. The text box component allows users to input one or more types of text, images, or videos, depending on their needs. The scroll view component displays a scrollable area containing text, images, or videos.

[0127] Masking components are used to clip the shape of their corresponding child components. Masking components can be text components, image components, button components, checkboxes, slider components, scrollbar components, dropdown lists, text box components, or scroll view components. Corresponding child components can be created for masking components, and these child components can be any of the following: text components, image components, button components, checkboxes, slider components, scrollbar components, dropdown lists, text box components, and scroll view components. Masking components have a primary shape; when a masking component is used to mask a corresponding child component, the shape of the displayed child component will be that primary shape. Effects components are used to add visual effects.

[0128] The following describes the pages provided in the embodiments of this application. The page is the page corresponding to the scene. The page corresponding to the scene refers to the page of the application displayed on the electronic device. There can be one or more pages. For example, for a game application, the scene can include any one of the following: main menu scene, game level scene, settings menu scene, and loading scene.

[0129] A scene can have one or more pages. For example, a loading scene can have only one page: the loading page. Similarly, a game level scene can have multiple pages, which may include at least two of the following: a page displaying the virtual scene, a page displaying information about virtual objects, and a page displaying scores. This is just an example; the specific settings can be adjusted based on actual usage.

[0130] In step 102, multiple canvas elements are filtered based on their priority to obtain the target canvas element to be used as the top-level canvas of the page.

[0131] Multiple canvas elements each have a corresponding priority. The priority of canvas elements can include inter-layer priority (Sorting Layer) and intra-layer priority (Order in Layer), which control the rendering order. Inter-layer priority, also known as sorting layer, controls the hierarchy of canvas elements in the rendering queue. By setting different sorting layers (inter-layer priorities), the rendering order of canvas elements can be determined. Inter-layer priority can be determined by the sorting layer list in the game engine (Unity). Layers further down the sorting layer list are rendered later and have higher inter-layer priority.

[0132] Intra-layer priority, also known as intra-layer sorting, allows for further refinement of the rendering order within the same sorting layer (i.e., under an inter-layer priority). The larger the value, the later the rendering order and the higher the intra-layer priority. This means that canvas elements rendered later will cover canvas elements rendered earlier.

[0133] Figure 3 Step 102 shown can be implemented in various ways, see [link to relevant documentation]. Figure 4 , Figure 4 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 2 The following is combined with Figure 4 The steps shown are for achieving Figure 3 The first implementation of step 102 is described below, wherein the priority of the canvas elements includes inter-layer priority and intra-layer priority used to control the rendering order.

[0134] In step 1021A, the candidate canvas element with the highest interlayer priority is determined from multiple canvas elements, and the number of candidate canvas elements is counted.

[0135] In some embodiments, each canvas element has an inter-layer priority for controlling the rendering order. Based on the inter-layer priority of each canvas element, a candidate canvas element with the highest inter-layer priority is determined from multiple canvas elements, and the number of candidate canvas elements is counted. When the number of candidate canvas elements is 1, the candidate canvas element can be used as the target canvas element. In this way, the target canvas element can be determined quickly.

[0136] In step 1022A, if the number is greater than 1, the candidate canvas element with the highest priority in the layer is determined as the target canvas element.

[0137] When the number of candidate canvas elements is greater than 1, the candidate canvas element with the highest intra-layer priority can be determined as the target canvas element. In other words, when there are multiple candidate canvas elements with the same inter-layer priority, the intra-layer priority can be further combined to determine the candidate canvas element with the highest intra-layer priority as the target canvas element. In this way, the precise target canvas element can be determined. The target canvas element is used to determine the target component, thereby generating training samples, which can facilitate the improvement of the accuracy of subsequent training sample generation.

[0138] See Figure 5 , Figure 5 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 3 The following is combined with Figure 5 The steps shown are for achieving Figure 3 The second implementation of step 102 is described below, wherein the priority of the canvas elements includes inter-layer priority and intra-layer priority used to control the rendering order.

[0139] In step 1021B, multiple canvas elements are sorted in descending order of interlayer priority to obtain a canvas element sequence.

[0140] In some embodiments, it can be determined whether the number of canvas elements is greater than a first quantity threshold. If the number of canvas elements is not greater than the first quantity threshold, that is, if the number of canvas elements is less than or equal to the first quantity threshold, multiple canvas elements can be sorted in descending order of interlayer priority to obtain a canvas element sequence.

[0141] For example, if there are three canvas elements: Canvas Element 1, Canvas Element 2, and Canvas Element 3, with Canvas Element 1 corresponding to layer priority 1, Canvas Element 2 to layer priority 2, and Canvas Element 3 to layer priority 3, then layer priority 1 is greater than layer priority 2, and layer priority 2 is greater than layer priority 3. Sort the three canvas elements according to their layer priority from highest to lowest to obtain the canvas element sequence: Canvas Element 1 - Canvas Element 2 - Canvas Element 3. This method allows for a precise determination of the canvas element sequence.

[0142] In some embodiments, when the number of canvas elements is greater than a first quantity threshold, a third canvas element with an interlayer priority greater than the first priority threshold is determined. Then, the third canvas elements are sorted according to the interlayer priority from largest to smallest to obtain a canvas element sequence. In this way, the canvas elements can be filtered first and then sorted, which can reduce the number of third canvas elements that need to be sorted and improve the efficiency of obtaining the canvas element sequence.

[0143] In some embodiments, when there is a layer priority corresponding to multiple fourth canvas elements, the multiple fourth canvas elements can be sorted in descending order of intra-layer priority, and the multiple fourth canvas elements and other canvas elements with inter-layer priorities can be sorted to obtain a canvas element sequence. In this way, an accurate canvas element sorting can be obtained.

[0144] In step 1022B, the area of ​​the page is obtained, and based on the area of ​​the page and the sequence of canvas elements, multiple canvas elements are filtered to obtain the target canvas element to be used as the top-level canvas of the page.

[0145] In some embodiments, the area of ​​the page can be obtained. If the page uses layout components, the width and height of the layout components can be obtained, thus giving the page area. The page area can also be preset, depending on the specific usage.

[0146] In some embodiments, after obtaining the area of ​​the page, multiple canvas elements can be filtered based on the page area and the sequence of canvas elements to obtain the target canvas element to be used as the top-level canvas of the page. In the embodiments of this application, the top-level canvas refers to the last canvas rendered in the page, that is, the topmost canvas used to display the page, and the top-level canvas can cover other canvases in the page.

[0147] In some embodiments, the first area ratio of the canvas elements is determined according to the order of the canvas elements in the canvas element sequence. The first area ratio can be the ratio of the area of ​​the canvas element to the area of ​​the page. That is, the ratio of the area of ​​each canvas element to the area of ​​the page is determined according to the order of the canvas elements in the canvas element sequence to obtain the first area ratio of the canvas elements.

[0148] After determining the first area ratio of the canvas elements, we can determine whether the first area ratio of the canvas elements is greater than a first ratio threshold according to the order of the canvas elements in the canvas element sequence. If the first canvas element with a first area ratio greater than the first ratio threshold is found, then the first canvas element in the canvas element sequence with a first area ratio greater than the first ratio threshold can be identified as the target canvas element. The first ratio threshold can be determined according to actual usage requirements; for example, the first ratio threshold can be 0.3.

[0149] In other words, according to the order of the canvas elements in the canvas element sequence, we can sequentially determine whether the first area ratio of the canvas elements is greater than the first ratio threshold. If the first area ratio of the canvas elements is less than or equal to the first ratio threshold, we can determine whether the first area ratio of the next canvas element is greater than the first ratio threshold, until the first canvas element with a first area ratio greater than the first ratio threshold appears. Then, the first canvas element in the canvas element sequence with a first area ratio greater than the first ratio threshold can be identified as the target canvas element.

[0150] For example, following the example of step 1021B, it can be determined whether the first area ratio of canvas element 1 is greater than the first ratio threshold. If the first area ratio of canvas element 1 is greater than the first ratio threshold, the judgment on canvas element 2 and canvas element 3 can be omitted, and canvas element 1 can be determined as the target canvas element.

[0151] If the first area ratio of canvas element 1 is less than or equal to the first ratio threshold, it can be determined whether the first area ratio of canvas element 2 is greater than the first ratio threshold. If the first area ratio of canvas element 2 is greater than the first ratio threshold, the judgment for canvas element 3 can be omitted, and canvas element 3 can be determined as the target canvas element.

[0152] If the first area ratio of canvas element 2 is less than or equal to the first ratio threshold, it can be determined whether the first area ratio of canvas element 3 is greater than the first ratio threshold. If the first area ratio of canvas element 3 is greater than the first ratio threshold, canvas element 3 can be identified as the target canvas element. In this way, by combining layer priority and the ratio of canvas elements to the page, the target canvas element among multiple canvas elements can be accurately determined. Compared with the method of determining the target canvas element only by combining layer priority, it can accurately filter out some canvases that are rendered later but not used for component recognition (such as canvases used to display special effects), so as to improve the accuracy of generating training samples in the future.

[0153] For steps 1021B-1022B, the control tree information can be obtained first. The control tree information is the way the page can represent the canvas and the components in the canvas. The control tree information is a tree structure. The control tree information includes multiple canvas elements of the page and the priority of the canvas elements. Then, based on the priority of each canvas element, multiple canvas elements can be filtered to obtain the target canvas element used as the top-level canvas of the page.

[0154] The second implementation can address both the scenario of managing pages by partitioning multiple canvas elements and the scenario of managing pages by layering multiple canvas elements. Both are reasonable. Through steps 1021B-1022B, multiple canvas elements or canvas elements with a priority greater than the first threshold can be sorted to obtain a canvas element sequence. Then, based on the page area and the canvas element sequence, the target canvas element to be used as the top-level canvas of the page can be accurately determined. Compared with the first implementation, this can further improve the accuracy of determining the target canvas element, which is equivalent to determining the precise top-level canvas. The target canvas element is used to determine the target component, thereby generating training samples. This facilitates the subsequent determination of the precise target component, thereby improving the accuracy of the generated training samples.

[0155] See Figure 6 , Figure 6 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 4 The following is combined with Figure 6 The steps shown are for achieving Figure 3 The third implementation of step 102 is described below, wherein the priority of the canvas element includes inter-layer priority and intra-layer priority used to control the rendering order.

[0156] In step 1021C, the canvas element with the highest priority among the multiple canvas elements is determined as the first canvas element.

[0157] In some embodiments, the canvas element with the highest inter-layer priority can be determined from multiple canvas elements. If there is only one canvas element with the highest inter-layer priority, it can be designated as the first canvas element. If there are multiple canvas elements with the highest inter-layer priority, the canvas element with the highest intra-layer priority can be determined and designated as the first canvas element. In this way, a precise first canvas element can be determined.

[0158] In step 1022C, based on the first canvas element and the priority of the canvas elements, other canvas elements among the multiple canvas elements are filtered to obtain the second canvas element.

[0159] In some embodiments, for other canvas elements, the first overlapping area between the other canvas elements and the first canvas element on the page can be determined, it can be determined whether the first overlapping area is less than a first area threshold, and it can be determined whether the distance between the first canvas element and other canvas elements is less than a first distance threshold.

[0160] If there are other canvas elements with a first overlapping area smaller than a first area threshold, and the distance between the first canvas element and other canvas elements is less than or equal to a first distance threshold, then the other canvas element with the highest priority is determined as the second canvas element. Here, the first overlapping area is the area of ​​overlap between the other canvas elements and the first canvas element on the page.

[0161] If other canvas elements have an overlap area smaller than the first area threshold, it indicates that multiple canvas elements manage corresponding pages in separate areas. If other canvas elements have an overlap area smaller than the first area threshold, and the distance between the first canvas element and other canvas elements is less than or equal to the first distance threshold, it indicates that the other canvas elements are adjacent to the first canvas element. Therefore, the other canvas element with the highest priority can be determined as the second canvas element. The first area threshold and the first distance threshold can be set according to actual usage requirements.

[0162] The distance between the first canvas element and other canvas elements is the distance between the first point in the first canvas element and the second point in other canvas elements. The first point and the second point can be the midpoint of the corresponding canvas element. The first point can also be any point in the fourth region of the first canvas element that is close to other canvas elements, and the second point can also be any point in the fifth region of other canvas elements that is close to the first canvas element.

[0163] If there are other canvas elements whose first overlapping area is greater than or equal to the first area threshold, it means that the other canvas elements overlap with the first canvas element. The first canvas element has a higher priority than the other canvas elements. In this case, the first canvas element is the top canvas of the corresponding area.

[0164] If there are other canvas elements with a first overlapping area smaller than the first area threshold, and the distance between the other canvas elements and the target area is greater than the second distance threshold, it means that the other canvas elements are not adjacent to the first canvas element. Further judgment can be made by combining the second overlapping area and the first area threshold.

[0165] If there are other canvas elements with a second overlapping area smaller than the first area threshold, and the distance between the other canvas elements and the target area is less than or equal to the second distance threshold, the other canvas element with the highest priority is determined as the second canvas element. The second overlapping area is the overlapping area between the other canvas element and the target area on the page, and the target area is the sum of the area of ​​the first canvas element on the page and the area of ​​the second canvas element on the page.

[0166] If other canvas elements have a second overlapping area greater than or equal to the first area threshold, it means these other canvas elements overlap with the first canvas element. The first canvas element has a higher priority than the other canvas elements. In this case, the first canvas element is the top-level canvas for the corresponding region. Alternatively, other canvas elements overlap with a determined second canvas element, and the determined second canvas element has a higher priority than the other canvas elements. In this case, the determined second canvas element is the top-level canvas for the corresponding region. Of course, these two cases can also be combined, where the first canvas element and the determined second canvas element are both the top-level canvases for their respective regions.

[0167] If there are other canvas elements with a second overlapping area smaller than the first area threshold, and the distance between the other canvas elements and the target area is greater than the second distance threshold, it means that the other canvas elements are not adjacent to the first canvas element and the determined second canvas element. Based on the determined second canvas element, the target area is re-determined, and then the second overlapping area and the first area threshold can be combined for further judgment.

[0168] In other words, for the case of managing pages corresponding to multiple canvas elements, after determining the first canvas element, the second canvas element can be determined multiple times until there are no other canvas elements with a second overlapping area smaller than the first area threshold. The number of second canvas elements can be one or more, thereby determining the canvas element with the highest priority corresponding to different areas of the page.

[0169] Each time a new second canvas element is identified, the target area can be updated. This allows for the determination of the new second canvas element by combining the second overlap area, the first area threshold, and the distances between other canvas elements and the target area. In this way, for pages with multiple canvas element partitions, the highest priority canvas element for each region can be accurately identified, meaning the second canvas element can be precisely determined.

[0170] In step 1023C, the first canvas element and the second canvas element are determined as the target canvas elements to be used as the top-level canvas of the page.

[0171] In some embodiments, the number of first canvas elements is 1, and the number of second canvas elements is one or more. The first canvas element and the second canvas element are determined as the target canvas elements used as the top-level canvas of the page. The number of target canvas elements is multiple. In this way, for the case of managing the corresponding page in multiple canvas element partitions, the highest priority canvas element corresponding to different areas can be accurately determined, that is, the precise target canvas element can be determined.

[0172] For ease of understanding, see Figure 7 , Figure 7 This is a schematic diagram of a page provided in an embodiment of this application. The following is in conjunction with... Figure 7 right Figure 6 The steps shown are illustrated with examples. Figure 7 The page contains three canvas elements: canvas element 701, canvas element 702, and canvas element 703.

[0173] Among canvas elements 701, 702, and 703, the canvas element 701 with the highest priority is determined as the first canvas element. For canvas elements 702 and 703 (i.e., other canvas elements), if the first overlap area between canvas elements 702 and 701 on the page is less than the first area threshold, and the distance between canvas elements 702 and 701 is less than or equal to the first distance threshold, then the canvas element 702 with the highest priority (other canvas elements) is determined as the second canvas element 1.

[0174] The first overlapping area of ​​canvas element 703 and canvas element 701 on the page is less than the first area threshold, and the distance between canvas element 702 and canvas element 701 is greater than the first distance threshold. Further judgment can be made by combining the second overlapping area and the first area threshold.

[0175] The second overlap area between canvas element 703 (other canvas elements) and the target area on the page is less than the first area threshold, and the distance between canvas element 703 and the target area is less than or equal to the second distance threshold. The target area is the sum of the areas of canvas element 701 and canvas element 702. Therefore, the canvas element 703 with the highest priority can be determined as the second canvas element 2.

[0176] Therefore, the first canvas element, the second canvas element 1, and the second canvas element 2 can be identified as the target canvas elements. In this way, for the case of managing pages with multiple canvas elements in different sections, the highest priority canvas element for each area can be accurately determined, that is, the precise target canvas element can be identified.

[0177] In some embodiments, assuming that canvas element a exists to the right of canvas element 703, the target area can be updated to the sum of the areas of canvas element 701, canvas element 702, and canvas element 703. This allows for determination of whether the second overlapping area of ​​canvas element a is less than a first area threshold, and whether the distance between canvas element a and the target area is less than or equal to a second distance threshold. Specific settings can be configured according to actual usage. That is, a target area can have one or more corresponding second canvas elements. After determining a new second canvas element, a new target area can be determined based on the determined second canvas element and the first canvas element, to further determine whether another second canvas element exists. In this way, for situations where multiple canvas elements are partitioned and managed across pages, the highest priority canvas element corresponding to different areas can be accurately determined, i.e., the precise target canvas element can be determined.

[0178] pass Figure 3 Step 102 shown can be implemented using one or more of the first to third implementation methods to determine the target canvas element. If the determined target canvas elements are different, the target canvas elements determined by different implementation methods can all be used as target canvas elements. In this way, one or more target canvas elements can be accurately determined. The target canvas elements are used to determine the target components, thereby generating training samples. This makes it easier to determine the accurate target components in the future, thereby improving the accuracy of the generated training samples.

[0179] See also Figure 3 In step 103, the target canvas element and the hierarchical relationship between multiple components in the target canvas element are determined.

[0180] In some embodiments, the number of target canvas elements can be one or more. For each target canvas element, the target canvas element may include one or more components. For ease of understanding, the target canvas elements and components are explained below with specific examples.

[0181] For example, see Figure 8 , Figure 8 This is a schematic diagram illustrating the hierarchical relationship provided in the embodiments of this application. For the target canvas element 801, the next level of the target canvas element 801 may include a task list 802 and a back button 806. That is, the upper level of the task list 802 and the back button 806 is the target canvas element 801. The task list 802 is a scrolling view component, and the back button 806 is a button component.

[0182] The next level of task list 802 can include task 803, task 804, and task 805. That is, the level above task 803, task 804, and task 805 is task list 802. Task 803, task 804, and task 805 are all text components. The next level of task 803 includes text 807. The level above text 807 is task 803. Text 807 is a text component.

[0183] In some embodiments, the hierarchical relationship between the target canvas element and its components can be obtained from the game engine's hierarchy view. In some embodiments, the hierarchical relationship between the target canvas element and its components can also be automatically obtained through pre-set code. In some embodiments, each node can be traversed using an interface provided by the game engine; each node corresponds to a canvas or component, thereby obtaining the hierarchical relationship between the target canvas element and its components. Different engines can be used for different application scenarios, and no specific limitation is made here. In this way, accurate hierarchical relationships can be obtained. These relationships are used to determine the target components, thereby generating training samples. Accurate hierarchical relationships facilitate subsequent determination of precise target components, improving the accuracy of the generated training samples.

[0184] See also Figure 3 In step 104, multiple components are filtered based on hierarchical relationships to obtain the target components that can be displayed on the page.

[0185] In some embodiments, the hierarchical relationship is represented by a tree structure comprising multiple nodes, with the target canvas element as the root node and components within the target canvas element as non-root nodes. For example, see [link to relevant documentation]. Figure 8 The hierarchical relationship is shown.

[0186] In some embodiments, step 104 can be implemented in the following way: if the number of target canvas elements is 1, the component corresponding to the child node of the root node can be determined as the target component that can be displayed on the page. For example, see Figure 8 The target canvas element 801 is the root node, and the task list 802 and the back button 806 are child nodes of the target canvas element 801. Therefore, the task list 802 and the back button 806 can be identified as target components that can be displayed on the page.

[0187] Among these, the target component that can be displayed on the page is called a control. In other words, the target component is the control that the user can observe on the page when it is displayed. Furthermore, the target component can be an interactive control, and the user can trigger the corresponding function of the interactive control through the corresponding interactive control. The root node only refers to the target canvas element, not the overall tree structure of the page.

[0188] In some embodiments, if there are multiple target canvas elements, and the overlap area between any two target canvas elements is less than a second area threshold, the component corresponding to the child node of the root node is determined as the target component that can be displayed on the page. That is, in the case of multiple canvas elements managing corresponding pages, where multiple target canvas elements do not interfere with each other, the component corresponding to the child node of the root node of each target canvas element can be determined as the target component that can be displayed on the page. In this way, different situations can be combined to quickly determine the target component, which is then used to generate training samples, further improving the efficiency of generating training samples.

[0189] In some embodiments, see Figure 9 , Figure 9 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 5 The following is combined with Figure 9 The steps shown are for Figure 3 Step 104 is illustrated below.

[0190] In step 1041, based on the hierarchical relationship, the component with a hierarchy level lower than the hierarchy threshold among the multiple components is determined as the first component.

[0191] The hierarchy can include multiple levels. For example, the hierarchy can include a first level, a second level, ..., an Nth level, where N is a positive integer greater than 2, and the specific level can be determined based on the actual situation. Based on the hierarchy, components with a level lower than a level threshold can be identified as the first component. The level threshold can be dynamically determined based on the number of levels, or it can be preset. There can be multiple first components. In this way, multiple components can be initially filtered based on the hierarchy. The first component with a level lower than the level threshold has a priority higher than a second priority threshold.

[0192] In some embodiments, when control tree information is obtained, the control tree information may include multiple components of the page, and there is an order between the components. For components at the same level, the lower the position of the component in the tree structure, the higher the priority of the component.

[0193] In some embodiments, for each level in the tree structure that is less than the level threshold, the first component can be traversed in a bottom-up order, which is equivalent to traversing the first component in descending order of priority.

[0194] During the traversal of the first component, if the first component is an image component, the transparency of the image component can be determined before executing step 1042. That is, when the first component is an image component, the transparency of the image component is determined. The transparency of the image component can be preset and can be any value between 0 and 1, where 0 represents complete transparency and 1 represents complete opacity.

[0195] In some embodiments, the transparency of an image component can also be determined by the transparent pixel value of the corresponding image. The transparent pixel value of each pixel of the image can be preset. The transparent pixel value is any value between 0 and 255, where 0 is completely transparent and 255 is completely opaque.

[0196] In some embodiments, a transparency threshold can be set for any value between 0 and 1. If the transparency of an image component is less than the transparency threshold, it indicates that the image component is a highly transparent image component.

[0197] In some embodiments, for any transparent pixel value between 0 and 255, if a preset number of transparent pixel values ​​are less than 255, it can be determined that the transparency of the image component is less than a transparency threshold. If the transparency of the image component is less than the transparency threshold, it indicates that the image component has a high degree of transparency. The preset number can be set according to actual usage requirements.

[0198] In some embodiments, for any transparent pixel value between 0 and 255, if a preset number of transparent pixel values ​​are less than a pixel value threshold, it can be determined that the transparency of the image component is less than the transparency threshold. If the transparency of the image component is less than the transparency threshold, it indicates that the image component has a high degree of transparency. The preset number can be set according to actual usage requirements.

[0199] In some embodiments, it can also be determined whether the area of ​​the image component is less than a third area threshold. If the area of ​​the image component is greater than or equal to the third area threshold, it indicates that the area of ​​the image to be displayed by the image component is large.

[0200] If the image component is highly transparent and the area of ​​the image to be displayed is large, it means that the image component is only used to display the corresponding image and not for interaction. Therefore, the image component can be filtered, that is, step 1042 is not performed on the image component.

[0201] If at least one of the following conditions is met, execution is triggered to filter multiple first components based on the hierarchy and area of ​​the first components to obtain the target component that can be displayed on the page, that is, to trigger step 1042:

[0202] 1. The transparency of the image component is greater than the transparency threshold. 2. The area of ​​the image component is less than the third area threshold. In other words, if the transparency of the image component is greater than the transparency threshold and the area of ​​the image component is less than the third area threshold, it indicates that the image component has low transparency and the area of ​​the image to be displayed is small, thus triggering step 1042.

[0203] If the transparency of an image component is greater than a transparency threshold and its area is greater than or equal to a third area threshold, it indicates that the image component has low transparency and the area of ​​the image it needs to display is large, thus triggering step 1042. This method allows for the filtering of image components to obtain precise target components, which are then used to generate training samples, improving the accuracy of the generated training samples.

[0204] If the transparency of the image component is less than or equal to the transparency threshold and the area of ​​the image component is less than the third area threshold, it indicates that the image component is a highly transparent image component and the area of ​​the image to be displayed by the image component is small, which can trigger the execution of step 1042.

[0205] In step 1042, multiple first components are filtered based on their hierarchy and area to obtain target components that can be displayed on the page.

[0206] In some embodiments, the plurality of first components include a plurality of levels. For any one first component, the first component at the next higher level is the parent component of the first component, and the first component at the next lower level is the child component of the first component.

[0207] For example, see Figure 8For task 803, the task list 802, which is one level above task 803, is the parent component of task 803; that is, task list 802 is the first component one level above task 803. The text 807, which is one level below task 803, is the child component of task 803; that is, text 807 is the first component one level below task 803.

[0208] In some embodiments, Figure 9 Step 1042 shown can be implemented in the following way: for the first component in each level, when the first component meets the target condition, the first component and its corresponding sub-components are determined as target components that can be displayed on the page.

[0209] The target conditions include three. The first target condition is: when the first component does not have a parent component, the second area ratio of the first component is greater than or equal to the second ratio threshold. The second area ratio is the ratio of the area of ​​the first component to the area of ​​the target canvas element.

[0210] The second objective condition is: when the first component has a parent component, the third area ratio of the parent component of the first component is greater than or equal to the second ratio threshold, where the third area ratio is the ratio of the area of ​​the parent component of the first component to the area of ​​the target canvas element.

[0211] The third objective condition is: when the first component has a parent component, the third area ratio of the parent component of the first component is less than the second ratio threshold, and the second area ratio of the first component is greater than or equal to the second ratio threshold.

[0212] In other words, each level can be processed in the order of hierarchy. If the level is the first level, for each first component in the first level, the first component does not have a parent component. The area ratio of the first component to the area of ​​the target canvas element where the first component is located can be determined. Then, it can be determined whether the second area ratio is less than the second ratio threshold. If the second area ratio of the first component is greater than or equal to the second ratio threshold, it can be determined that the first component meets the target condition. Therefore, the first component and its corresponding child components can be determined as target components that can be displayed on the page.

[0213] For example, see Figure 8 For components, task list 802 and back button 806 are the first components in the first level. Neither task list 802 nor back button 806 has a parent component. If the second area ratio of task list 802 is greater than or equal to the second ratio threshold, task list 802 and its child components (task 803, task 804, task 805 and text 807) can be identified as target components that can be displayed on the page.

[0214] If the second area ratio of the first component is less than the second ratio threshold, the first component in the next level of the first level can be judged. The first component in the next level of the first level has a first component that has been determined as the target component and a first component that has not been determined as the target component. For the first component that has been determined as the target component, the first component has a parent component. The third area ratio of the parent component of the first component is greater than or equal to the second ratio threshold. The third area ratio is the area ratio of the parent component of the first component to the target canvas element.

[0215] For each first component that is not identified as a target component, the first component has a parent component (i.e., the first component at the first level), and the third area ratio of the parent component of the first component is less than the second ratio threshold. It can be determined whether the second area ratio of the first component is less than the second ratio threshold. If the second area ratio of the first component is greater than or equal to the second ratio threshold, the first component and its corresponding child components can be identified as target components that can be displayed on the page.

[0216] For example, in see Figure 8 If the second area ratio of task list 802 is less than the second ratio threshold, for the first component in the next level of the first level, i.e. for task 803, task 804 and task 805, it can be determined whether the second area ratio of task 803 is less than the second ratio threshold. If the second area ratio of task 803 is greater than or equal to the second ratio threshold, the first component (task 803) and the corresponding sub-component (text 807) of the first component are determined as target components that can be displayed on the page.

[0217] For task 804, it can be determined whether the second area ratio of task 804 is less than the second ratio threshold. If the second area ratio of task 804 is greater than or equal to the second ratio threshold, task 804 can be identified as a target component that can be displayed on the page. The description for task 805 can be equivalently replaced with that for task 804, and will not be repeated here.

[0218] If the second area ratio of the first component is greater than or equal to the second ratio threshold, the first component in the next level of the second level can be judged. The specifics can be equivalently replaced according to the above description, which will not be elaborated here.

[0219] For example, if the second area ratios of tasks 803, 804, and 805 are all less than the second ratio threshold, it can be determined whether the second area ratio of text 807 is less than the second ratio threshold. If the second area ratio of text 807 is greater than or equal to the second ratio threshold, text 807 is identified as a target component that can be displayed on the page. In this way, by combining the hierarchical relationship and the area of ​​the first component, a precise target component can be determined. This target component is used to generate training samples, which can improve the accuracy of generating training samples.

[0220] In some embodiments, Figure 9 Step 1042 shown can be implemented in the following way: multiple first components include a first level and a second level. For the first component of the first level, a second area ratio of the area of ​​the first component to the area of ​​the target canvas element where the first component is located can be determined. Then, it can be determined whether the second area ratio is less than a second ratio threshold. If the second area ratio of the first component is greater than or equal to the second ratio threshold, it can be determined that the first component meets the target condition. The first component and its sub-components can be determined as target components that can be displayed on the page. The second area ratio is the area ratio of the first component to the target canvas element. The second level of the sub-component is the next level of the first level. The second level of the sub-component belongs to the first level.

[0221] If the second area ratio of the first component is less than the second ratio threshold, for the first component in the second level that is not determined as a target component, the second area ratio of the area of ​​the first component to the area of ​​the target canvas element where the first component is located can be determined. Then, it can be determined whether the second area ratio is less than the second ratio threshold. If the second area ratio of the first component is greater than or equal to the second ratio threshold, the first component in the second level is determined as the target component that can be displayed on the page.

[0222] If the second area ratio of the first component is less than the second ratio threshold, the judgment can stop if there is no next level in the second level. If there is a next level in the second level, the judgment can continue for the first component of the next level. For details, please refer to the explanation of the judgment for the second level and make equivalent substitutions, which will not be elaborated here. In this way, the precise target component can be determined by combining the hierarchical relationship and the area of ​​the first component. The target component is used to generate training samples, which can improve the accuracy of generating training samples.

[0223] Through steps 1041-1042, the first component can be screened out first. Then, by combining the level and area of ​​the first component, the first component can be further screened to determine the precise target component. The target component is used to generate training samples, which can improve the accuracy of generating training samples.

[0224] Continue to participate Figure 3 In step 105, the position of the target component in the target canvas element is determined, and sample labels are constructed based on the position. The page image corresponding to the page is labeled to generate training samples for training the component recognition model.

[0225] In some embodiments, the position of the target component in the target canvas element can be determined. When the rendering mode is the first rendering mode, the position of the target component in the target canvas element can be obtained by managing the component (RectTransform).

[0226] When the rendering mode is the second or third rendering mode, the world coordinates of the target component in the world coordinate system can be obtained. In some embodiments, the world coordinates of the target component in the world coordinate system (WorldCorners) can be obtained through the management component (RectTransform).

[0227] After obtaining the world coordinates, they can be transformed from the world coordinate system to the view coordinate system to obtain the view coordinates. The view coordinate system is a coordinate system centered on the camera. The view matrix of the camera can be obtained, and then the world coordinates can be multiplied by the view matrix to obtain the view coordinates.

[0228] After obtaining the view coordinates, the view coordinates can be converted into clipping coordinates for clipping the target component. The clipping coordinates are the coordinates after the lens projection matrix transformation, which can be used to determine whether the camera is located within the view frustum. The view coordinates can be multiplied by the projection matrix to obtain the clipping coordinates.

[0229] The clipping coordinates are four-dimensional vectors. The projection matrix includes a perspective projection matrix and an orthographic projection matrix. The perspective projection matrix is ​​used by a perspective camera to project points in three-dimensional space onto a two-dimensional plane while preserving depth information. The orthographic projection matrix is ​​used by an orthographic camera to directly map points in three-dimensional space onto a two-dimensional plane without considering perspective effects.

[0230] Converting view coordinates to clipping coordinates for cropping the target component can also be described as mapping view coordinates to a 2D page, resulting in clipping coordinates for cropping the target component. When the projection matrix is ​​a perspective projection matrix, the denominator of the perspective division is related to the z-axis coordinate. When the projection matrix is ​​an orthographic projection matrix, the denominator of the perspective division can be 1.

[0231] After obtaining the clipping coordinates, they can be mapped to the display area on the page to obtain the position of the target component within the target canvas element. In some embodiments, the clipping coordinates include the coordinates after projection transformation and the denominator of perspective division. Perspective division can be used to obtain normalized coordinates, where perspective division is performed by dividing each coordinate value in the projection transformation by the denominator of perspective division to obtain normalized coordinates.

[0232] After obtaining the normalized coordinates, the normalized coordinates can be converted into screen coordinates through viewport transformation. This is equivalent to mapping the clipping coordinates to the display area on the page. If the target component is a rectangle, the screen coordinates are the coordinates of the four vertices of the rectangle. The coordinates of the diagonal of the rectangle can be used to determine the position of the target component in the target canvas element.

[0233] For example, in the gaming field, for a "backpack component," which is a rectangle, the coordinates of the top-left corner of the "in-game presentation component" are (339, 24), and the coordinates of the bottom-right corner are (426, 96). In some embodiments, the coordinates of the midpoint of the rectangle can also be obtained; for example, for the "backpack component," the coordinates of the midpoint of the "in-game presentation component" are (382, 60). In this way, the precise position of the target component within the target canvas element can be obtained. This position is used to generate training samples, which can improve the accuracy of the generated training samples.

[0234] After obtaining the position of the target component within the target canvas element, sample labels can be constructed based on the position to annotate the corresponding page image and generate training samples for training the component recognition model.

[0235] In some embodiments, the page image is the input of the component recognition model, and the output can be the position of the target component in the target canvas element. For example, it can be the coordinates of the diagonal of the rectangle corresponding to the target component, or, for example, it can be the rectangle marked by the coordinates of the diagonal in the page image.

[0236] In some embodiments, the name of the target component can be obtained, and sample labels can be constructed based on the location and the name of the target component to annotate the page image corresponding to the page, generating training samples for training the component recognition model. Accordingly, the page image of the page is the input of the component recognition model, and the output can be the position of the target component in the target canvas element and the name of the target component.

[0237] The training sample generation method provided in this application embodiment can obtain multiple canvas elements on a page, each with a corresponding priority. Each canvas element includes multiple components. Based on the priority of each canvas element, the multiple canvas elements are filtered to obtain the target canvas element used as the top-level canvas of the page. This is equivalent to performing a first filtering based on the priority of the canvas elements to obtain the target canvas element. The hierarchical relationship between the target canvas element and its multiple components is determined. Based on the hierarchical relationship, the multiple components are filtered to obtain the target components that can be displayed on the page. This is equivalent to performing a second filtering based on the hierarchical relationship to obtain the target components. Then, the position of the target component in the target canvas element can be determined, and sample labels are constructed based on the position to annotate the corresponding page image, generating training samples for training the component recognition model.

[0238] As can be seen, in the training sample generation method provided in this application embodiment, the canvas elements can be first filtered according to their priority to obtain target canvas elements. Then, based on the hierarchical relationship, multiple components in the target canvas elements are filtered a second time to obtain target components. In other words, for the canvas elements on the page, two different filters can be performed to obtain accurate target components that can be displayed on the page, without requiring the user to manually mark the components on the page. This reduces the time required to determine the target components on the page and improves the efficiency of determining the target components on the page. Furthermore, sample labels can be constructed based on the position of the target components to generate training samples, thereby reducing the time required to generate training samples, improving the efficiency of generating training samples, and improving the accuracy of generating training samples, thus solving technical problem 1.

[0239] In some embodiments, the number of target components is multiple, and the multiple target components include a second component for displaying text. The second component can be a text component or a text box component, which can be set according to the actual usage request.

[0240] See Figure 10 , Figure 10 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 6 The following is combined with Figure 10 The steps shown illustrate the method for generating training samples for training the collision recognition model in the embodiments of this application. Figure 10 The steps shown are in Figure 3 This is executed after the "Determine the position of the target component in the target canvas element" shown.

[0241] In step 1001, if the target component includes a third component, the collision relationship between the second component and the third component on the page is determined based on the positions of the second component and the third component.

[0242] In some embodiments, it can be first determined whether a third component is included among multiple target components based on the second component. It can be determined whether a third component exists whose distance from a third point of the second component is less than a third distance threshold. The third point of the second component can be the midpoint of the second component, or any point in a first region of the second component close to the third component. A fourth point exists in the third component; this fourth point can correspond to the midpoint of the third component, or any point in a second region of the third component close to the second component. The distance between the third component and the second component is less than the third distance threshold.

[0243] In some embodiments, components adjacent to the second component can be determined first. Components adjacent to the second component are those belonging to the same target canvas element and not separated from the second component by other components. The distance between the fifth point of the component adjacent to the second component and the third point of the second component is determined. The fifth point of the component adjacent to the second component can be the midpoint of the component adjacent to the second component, or any point in the third region of the component adjacent to the second component. If the distance is less than a third distance threshold, the component adjacent to the second component is the third component.

[0244] When the target component includes a third component, the collision relationship between the second and third components on the page is determined based on the positions of the second and third components. In some embodiments, both the second and third components are rectangular in shape. The position of the second component includes a first angular coordinate and a second angular coordinate, and the position of the third component includes a third angular coordinate and a fourth angular coordinate. The first, second, third, and fourth angular coordinates are in the same two-dimensional coordinate system, and the positions of the first and third angular coordinates within the rectangle correspond to the positions of the second and fourth angular coordinates within the rectangle.

[0245] In some embodiments, if the left boundary of the rectangle of the second component is smaller than the right boundary of the rectangle of the third component, and the right boundary of the rectangle of the second component is larger than the left boundary of the rectangle of the third component, then the collision relationship is determined to be that the second component and the third component collide on the page.

[0246] In some embodiments, if the x-coordinate of the first angular coordinate is greater than the x-coordinate of the third angular coordinate, and the x-coordinate of the second angular coordinate is less than the x-coordinate of the fourth angular coordinate, then the collision relationship is determined to be that the second component and the third component collide on the page.

[0247] In some embodiments, if the left boundary of the rectangle of the second component is smaller than the right boundary of the rectangle of the third component, the right boundary of the rectangle of the second component is larger than the left boundary of the rectangle of the third component, the top boundary of the rectangle of the second component is not equal to the bottom boundary of the rectangle of the third component, and the bottom boundary of the rectangle of the second component is not equal to the top boundary of the rectangle of the third component, it indicates that the second component and the third component overlap, that is, the second component and the third component collide, that is, the text displayed in the second component exceeds the border, i.e., there is text over-border.

[0248] In some embodiments, if the x-coordinate of the first angular coordinate is greater than the x-coordinate of the third angular coordinate, the x-coordinate of the second angular coordinate is less than the x-coordinate of the fourth angular coordinate, the y-coordinate of the first angular coordinate is not equal to the y-coordinate of the third angular coordinate, and the y-coordinate of the second angular coordinate is not equal to the y-coordinate of the fourth angular coordinate, then the collision relationship is determined to be a collision between the second component and the third component on the page. In this way, the existence of a collision can be determined by combining the component coordinates without manual determination, enabling automated collision relationship determination and improving the efficiency and accuracy of collision relationship determination.

[0249] In some embodiments, if the left boundary of the rectangle of the second component is greater than (or equal to) the right boundary of the rectangle of the third component, and the right boundary of the rectangle of the second component is less than (or equal to) the left boundary of the rectangle of the third component, then the collision relationship is determined to be that the second component and the third component do not collide on the page.

[0250] In some embodiments, if the x-coordinate of the first angular coordinate is less than (or equal to) the x-coordinate of the third angular coordinate, and the x-coordinate of the second angular coordinate is greater than (or equal to) the x-coordinate of the fourth angular coordinate, then the collision relationship is determined to be that the second component and the third component do not collide on the page.

[0251] If there is no collision between the second and third components of the collision relationship representation, the results of no collision and the corresponding image of the page can be stored to obtain training samples for training the collision recognition model. Among them, the training samples corresponding to the absence of collision between the second and third components of the collision relationship representation are positive samples. Users can determine the number of positive samples according to their actual usage needs.

[0252] In step 1002, if there is a collision between the second component and the third component in the collision relationship characterization, the target position is determined based on the position of the second component and the position of the third component.

[0253] In some embodiments, the difference between the ordinate of the second angular coordinate and the ordinate of the fourth angular coordinate is determined. If the difference is greater than or equal to a difference threshold, the second angular coordinate and the fourth angular coordinate are determined as the coordinates of the target position. The second angular coordinate and the fourth angular coordinate are the coordinates corresponding to the diagonal of the rectangle formed by the union of the third component and the fourth component.

[0254] When the difference is less than the difference threshold, the target position can be determined based on the first, second, third, and fourth angular coordinates. In some embodiments, the ordinate of the first and third angular coordinates can be determined as the ordinate of the target position, and the abscissa of the second and fourth angular coordinates can be determined as the abscissa of the target position.

[0255] If the ordinate of the first angular coordinate is greater than the ordinate of the third angular coordinate, and the abscissa of the second angular coordinate is less than the abscissa of the fourth angular coordinate, the ordinate of the first angular coordinate and the abscissa of the second angular coordinate can be combined to obtain one angular coordinate of the target position. The ordinate of the third angular coordinate and the abscissa of the fourth angular coordinate can be combined to obtain the other angular coordinate of the target position.

[0256] In other words, the target location can be determined by combining the largest ordinate with the smallest abscissa, and vice versa. This means determining the coordinates corresponding to the diagonal of the rectangle formed by the union of the third and fourth components. By combining the difference between the ordinates of the second and fourth angles and the relationship with a threshold difference, different methods can be used to determine the target location, thus improving the accuracy of target location determination.

[0257] In step 1003, sample labels are constructed based on the target location, and the page image corresponding to the page is labeled to obtain training samples for training the collision recognition model.

[0258] In some embodiments, the page image is the input to the component recognition model, and the output can be the position of the rectangle formed by the union of the colliding components when collisions exist.

[0259] Through steps 1001-1003, after determining the position of the target component, it is further determined whether there is a collision between the second and third components, thereby generating training samples for training the collision recognition model. This can reduce the time required to generate training samples, improve the efficiency of generating training samples, and improve the accuracy of generating training samples, thus solving technical problem 1.

[0260] To facilitate understanding of the training sample generation method provided in this application embodiment, the following description, in conjunction with specific examples, illustrates the training sample generation method provided in this application embodiment. The training sample generation method provided in this application embodiment can quickly collect effective control information (i.e., target component and the coordinates of the target component) and abnormal information of text super-boundaries (i.e., second component, third component, and target area) on the page. Thus, it can automatically generate training samples for the corresponding component recognition model by combining the effective control information and automatically generate training samples for the corresponding collision detection model by combining the abnormal information of the text super-boundaries.

[0261] See Figure 11 , Figure 11 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 7 The following is combined with Figure 11 The steps describe how to determine the top-level canvas (target canvas element). In step 1101, multiple canvas elements on the page are obtained. For step 1101, control tree information can be obtained. The control tree information corresponds to a tree structure and includes multiple canvas elements on the page and the priority of the canvas elements.

[0262] In step 1102, multiple canvas elements are sorted according to their layer priority from highest to lowest to obtain a canvas element sequence. For step 1102, please refer to [the documentation for...]. Figure 5 The explanation of step 1021B shown will not be repeated here.

[0263] In step 1103, based on the canvas element sequence, a first area ratio of the area of ​​the canvas element to the area of ​​the page is determined. For step 1103, the first area ratio of the first canvas element in the canvas element sequence to the area of ​​the page can be determined first, and then step 1104 can be executed.

[0264] In step 1104, it is determined whether the first area ratio is greater than the first ratio threshold. If the first area ratio is greater than the first ratio threshold, step 1105 is executed; if the first area ratio is not greater than the first ratio threshold, step 1103 is executed.

[0265] If the first area ratio of the first canvas element is greater than the first ratio threshold, other canvas elements in the canvas element sequence may not be processed. In step 1105, the first canvas element whose first area ratio is greater than the first ratio threshold is determined as the target canvas element.

[0266] If the first area ratio of the first canvas element is less than or equal to the first ratio threshold, step 1103 can be executed. In step 1103, according to the arrangement order of each canvas element in the canvas element sequence, for the canvas element located next to the first canvas element, the first area ratio of the area of ​​the canvas element to the area of ​​the page is determined, and then step 1104 can be executed. For details, please refer to the above description of the first canvas element and make equivalent substitutions for the corresponding content. It will not be repeated here.

[0267] Combination Figure 11 As can be seen, in the training sample generation method provided in this application embodiment, for different canvas elements, the first area ratio of each canvas element can be determined sequentially according to the priority of the canvas elements from large to small to see if it is greater than the first ratio threshold. If it is greater than the threshold, the corresponding canvas element is determined as the target canvas element to be used as the top-level canvas of the page, and the determination of the remaining canvas elements is stopped. In this way, the precise target canvas element can be determined.

[0268] See Figure 12 , Figure 12 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 8 The following is combined with Figure 12 The steps describe how to determine the top-level component (target component). In step 1201, multiple components in the target canvas element are obtained. Multiple components can exist in the target canvas element.

[0269] In step 1202, multiple components are traversed from bottom to top. Step 1202 is equivalent to traversing multiple components in descending order of component priority, thereby executing step 1203 or step 1206 for each component, as explained in detail below.

[0270] If the component is an image component, step 1206 can be executed. In step 1206, it is determined whether the transparency is 0. If the transparency is 0, step 1208 can be executed. If the transparency is not 0, step 1207 can be executed.

[0271] In step 1207, it is determined whether the transparency is less than a transparency threshold and whether the area of ​​the image component is greater than a third area threshold. If the transparency is less than the transparency threshold and the area of ​​the image component is greater than the third area threshold, it indicates that the image component is a semi-transparent large image, and step 1208 can be executed, in which the image component is filtered. Here, semi-transparency does not mean that the transparency is 50%, but rather that the transparency of the image component is less than the transparency threshold.

[0272] If the transparency is greater than the transparency threshold and / or the area of ​​the image component is greater than the third area threshold, it indicates that the image component is not a semi-transparent large image, and step 1203 can be executed. Step 1203 can also be executed if the component is not an image component.

[0273] In step 1203, a second area ratio between the area of ​​the component and the area of ​​the target canvas element is determined. Regarding step 1203, in this embodiment, step 1203 can be performed on components other than the filtered image components, or it can be performed on a first component other than the filtered image components. The first component can be referred to the above description and will not be repeated here.

[0274] In step 1204, it is determined whether the second area ratio is greater than the second ratio threshold. If the second area ratio is greater than the second ratio threshold, step 1205 can be executed. In step 1205, the component can be identified as the target component, and then the process can return to step 1203. If the second area ratio is less than or equal to the second ratio threshold, step 1203 can be executed, which is equivalent to executing step 1203 for the next component.

[0275] Combination Figure 12 As can be seen, in the training sample generation method provided in this application embodiment, the second area ratio of each component can be judged in descending order of component priority to determine whether it is greater than the second ratio threshold. If it is greater, the corresponding component is determined as the target component to be displayed on the page. In this way, the accurate target component can be determined.

[0276] See Figure 13 , Figure 13 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 9 In step 1301, the information of the target component is obtained. This information refers to the target component's attribute information within the game engine.

[0277] In step 1302, it is determined whether the target component includes a first attribute or a second attribute. The first attribute is the event system attribute (IEventSystemHandler), and the second attribute is the text attribute (UI.Text). If the target component includes the first attribute, step 1303 can be executed, in which the target component is determined to be the first target component. The first target component is a control that does not display text but is interactive.

[0278] If the target component includes a second attribute, or if the target component includes both a first attribute and a second attribute, step 1304 can be executed, in which the target component is determined to be a second target component. The second target component is the component that displays the text. If the target component does not include the first attribute and the second attribute, the target component can be filtered out.

[0279] In step 1305, the positions of the first target component and the second target component are determined. In step 1306, a training set is generated. Based on steps 1305 and 1306, the positions of the target components within the target canvas elements can be determined, and sample labels are constructed based on these positions. The corresponding page images are then labeled to generate training samples (i.e., the training set) for training the component recognition model.

[0280] See Figure 14 , Figure 14 This is a flowchart illustrating the training sample generation method provided in the embodiments of this application. Figure 10 In step 1401, information about the target component is obtained. This information includes the rendering mode. In step 1402, it is determined whether the rendering mode is empty. If the rendering mode is empty, the corresponding target component can be filtered out. If the rendering mode is not empty, step 1403 is executed.

[0281] In step 1403, the world coordinates of the target component in the world coordinate system are obtained. For step 1403, the world coordinates (WorldCorners) of the target component in the world coordinate system can be obtained through the management component (RectTransform). The management component is used to manage the position, size, and layout of the component.

[0282] In step 1404, world coordinates are converted to screen coordinates. For step 1404, the world coordinates can be transformed from the world coordinate system to the view coordinate system to obtain view coordinates. The view coordinates are then converted into clipping coordinates for clipping the target component. The clipping coordinates are mapped to the display area on the page to obtain the position of the target component in the target canvas element.

[0283] In some embodiments, world coordinates can be multiplied by a view matrix to obtain view coordinates, and view coordinates can be multiplied by a projection matrix to obtain clipping coordinates. Based on the viewport matrix, the clipping coordinates are converted into screen coordinates. The component can be a rectangle, and the screen coordinates are the coordinates corresponding to the four vertices of the rectangle. The coordinates of the diagonal of the rectangle can be determined as the position of the target component in the target canvas element.

[0284] For example, in the gaming field, for an "in-game presentation component," the "in-game presentation component" is a rectangle, with the coordinates of its top-left corner being (118, 840) and its bottom-right corner being (268, 960). In some embodiments, the coordinates of the rectangle's midpoint can also be obtained; for example, for the "in-game presentation component," the midpoint coordinates are (193, 900).

[0285] The following example illustrates the training samples used to train the component recognition model. The training samples may include the page image of the "intra-visual component", the coordinates of the upper left corner of the "intra-visual component" in the page image (118, 840), and the coordinates of the lower right corner of the "intra-visual component" in the page image (268, 960).

[0286] In some embodiments, after obtaining the second target component, that is, after obtaining the second component used to display text, if there is a third component adjacent to the second component among multiple target components, the collision relationship between the second component and the third component on the page can be determined based on the position of the second component and the position of the third component. Both the second component and the third component are rectangles, and the collision relationship between the second component and the third component on the page can be detected by a rectangle collision algorithm.

[0287] When there is a collision between the second and third components in the collision relationship representation, the target position is determined based on the positions of the second and third components. Sample labels are constructed based on the target positions, and the corresponding page images are labeled to obtain training samples for training the collision recognition model.

[0288] In some embodiments, if the left boundary of the rectangle of the second component is smaller than the right boundary of the rectangle of the third component, the right boundary of the rectangle of the second component is larger than the left boundary of the rectangle of the third component, the upper boundary of the rectangle of the second component is not equal to the lower boundary of the rectangle of the third component, and the lower boundary of the rectangle of the second component is not equal to the upper boundary of the rectangle of the third component, it indicates that the second component and the third component overlap, that is, the second component and the third component collide.

[0289] For example, see Figure 15 , Figure 15 This is a schematic diagram of the collision relationship provided in the embodiments of this application. Figure 15 The image shows a second component 1501 and a third component 1504. The position of the second component 1501 includes a first angular coordinate 1502 and a second angular coordinate 1503, and the position of the third component 1504 includes a third angular coordinate 1505 and a fourth angular coordinate 1506.

[0290] The first angular coordinate 1502 is (x1, y1), the second angular coordinate 1503 is (x2, y2), the third angular coordinate 1505 is (x3, y3), and the fourth angular coordinate 1506 is (x4, y4). If x3 is less than x2, x4 is greater than x1, y1 is not equal to y4, and y2 is not equal to y3, it indicates that the second component 1501 and the third component 1504 are in a collision.

[0291] See Figure 16 , Figure 16 This is a schematic diagram of the target area provided in the embodiments of this application, which is intended to... Figure 15 The first angular coordinate 1502 of the second component 1501 and the fourth angular coordinate 1506 of the third component 1504 can be used as the coordinates of the target area 1601 (i.e., the area corresponding to the dashed box).

[0292] This application embodiment can obtain training samples for training component recognition models and collision recognition models without manual labeling. It is equivalent to automatically collecting training samples for recognition components and training samples for text over-boundary anomalies (corresponding to collision recognition). This application embodiment provides a way to quickly collect page component information, anomaly information, and the regions corresponding to the anomalies, thereby generating corresponding training samples. This greatly reduces the time required to generate small training samples. For example, in the game industry, in related technologies, manually labeling training samples for game page components takes more than 15 days. Using the training sample generation method provided in this application embodiment, training samples for training component recognition models can be generated within one hour.

[0293] The component identification model provided in the embodiments of this application is described below. See also... Figure 17 , Figure 17 This is a schematic diagram of the component recognition model provided in the embodiment of this application. The component recognition model includes an input layer 1701, a feature extraction layer 1702, a feature processing layer 1703, and an output layer 1704. The input layer 1701 is connected to the feature extraction layer 1702, the feature extraction layer 1702 is connected to the feature processing layer 1703, and the feature processing layer 1703 is connected to the output layer 1704.

[0294] In some embodiments, page images from training samples used to train the component recognition model can be input to the input layer. The input layer can preprocess the page images, such as adjusting the image size, normalizing pixel values, etc., which can be set according to the actual situation.

[0295] The preprocessed page image is input into the feature extraction layer, which can be a convolutional layer. The convolutional kernel performs convolution and feature extraction on the preprocessed page image. Each convolutional kernel slides on the preprocessed page image. By calculating the sum of the product of the convolutional kernel and the corresponding pixel values ​​of the local region of the preprocessed page image, the image features of the preprocessed page image are extracted.

[0296] Image features are input into the feature processing layer, which can be a pooling layer or a fully connected layer. The pooling layer is used to downsample the image features, reducing the dimension of the feature map corresponding to the image features, thereby reducing the amount of computation and improving the robustness of the component recognition model.

[0297] The downsampled image features can be input into a fully connected layer. The role of the fully connected layer is to integrate the downsampled image features. Through weight and bias operations, it calculates the probability value of each component corresponding to the downsampled image features. The fully connected layer can output the probability of the component and its corresponding coordinate information. The output layer can output the predicted component with the highest probability and its corresponding predicted coordinate information.

[0298] The parameters of the component recognition model can be adjusted based on the differences between the predicted component and the target component, as well as the differences between the target component's position within the target canvas elements and the predicted coordinate information, thus obtaining the component recognition model. The output layer can also output only preset coordinate information.

[0299] For example, see Figure 18 , Figure 18 This is a schematic diagram of the output of the component identification model provided in the embodiments of this application. Figure 1 The page image 1801 can be output to the component recognition model, thereby outputting a solid-line frame 1802 marked on the page image 1801. The solid-line frame 1802 is used to mark the position of the email component. The coordinates 1803 of the email component can also be output, which are (814, 13) and (854, 97). (814, 13) and (854, 97) are the coordinates of the diagonal of the rectangle corresponding to the solid-line frame 1802.

[0300] The collision recognition model provided in the embodiments of this application is described below. See also... Figure 19 , Figure 19 This is a schematic diagram of the collision recognition model provided in the embodiment of this application. The component recognition model includes an input layer 1901, a feature extraction layer 1902, a collision recognition layer 1903, and an output layer 1904. The input layer 1901 is connected to the feature extraction layer 1902, the feature extraction layer 1902 is connected to the collision recognition layer 1903, and the collision recognition layer 1903 is connected to the output layer 1904.

[0301] In some embodiments, page images from training samples used to train the collision recognition model can be input into the input layer. The input layer can preprocess the page images, such as adjusting the image size, normalizing pixel values, etc., which can be set according to the actual situation.

[0302] The preprocessed page image is input into the feature extraction layer, which can be a convolutional layer. The convolutional kernel performs convolution and feature extraction on the preprocessed page image. Each convolutional kernel slides on the preprocessed page image. By calculating the sum of the product of the convolutional kernel and the corresponding pixel values ​​of the local region of the preprocessed page image, the image features of the preprocessed page image are extracted.

[0303] Image features are input into a collision detection layer, which determines whether a collision exists between the second and third components and calculates the coordinates of the collision region. The collision detection layer can employ any of the following methods to determine whether a collision exists between the second and third components: image-space-based collision detection algorithms, geometric-space-based collision detection algorithms, and deep learning methods.

[0304] Among these methods, image-space-based collision detection algorithms project components onto a two-dimensional plane using graphics hardware and determine collisions based on the intersection of the projections. Geometric-space-based collision detection algorithms wrap components within axis-aligned rectangles and detect collisions by checking if these rectangles intersect. Deep learning-based methods classify image features and then determine the presence of collisions based on the similarity between the image features and the features of each category.

[0305] The output layer can output the results of the collision detection layer. In the case of a collision, it can output the predicted coordinates of the rectangle corresponding to the union of the second and third components, that is, the predicted coordinates of the minimum bounding rectangle of the second and third components.

[0306] The parameters of the collision recognition model can be adjusted based on the difference between the predicted result and the label result (corresponding to whether a collision exists), as well as the difference between the predicted coordinates and the target position, so as to obtain the collision recognition model.

[0307] For example, see Figure 20 , Figure 20 This is a schematic diagram of the output of the collision recognition model provided in the embodiments of this application. Figure 1 If there is a collision 2001 between the second and third components of the displayed text, the collision recognition model can output a page image marked with a dashed box 2002.

[0308] For example, see Figure 21 , Figure 21 This is a schematic diagram of the output of the collision recognition model provided in the embodiments of this application. Figure 2The collision recognition model can output a page image marked with a dashed bounding box 2101. For example, see... Figure 22 , Figure 22 This is a schematic diagram of the output of the collision recognition model provided in the embodiments of this application. Figure 3 If there is a collision between the second component 2201 and the third component 2202, a dashed box 2203 can be marked in the output page image to indicate that there is a collision between the second component 2201 and the third component 2202. Figures 20-21 The dashed box in the model is determined by the coordinates output by the collision recognition model.

[0309] See Figure 23 , Figure 23 This is a schematic diagram of the output of the component identification model provided in the embodiments of this application. Figure 2 Without determining the target component based on hierarchical relationships and component area, the number of target components is large. Figure 23 The target component 2301 is only marked in the image. It can be seen that the target component 2301 is not an interactive control. Therefore, the accuracy of determining the target component without considering the hierarchical relationship and the area of ​​the component is low.

[0310] See Figure 24 , Figure 24 This is a schematic diagram of the output of the component identification model provided in the embodiments of this application. Figure 3 When the target component is determined by the hierarchical relationship and the area of ​​the component, the number of target components is relatively small compared to... Figure 23 In fewer cases, it is possible to obtain more precise target components. Figure 24 Only the target component 2401 is marked in the text.

[0311] The training sample generation method provided in this application embodiment can obtain multiple canvas elements on a page, each with a corresponding priority. Each canvas element includes multiple components. Based on the priority of each canvas element, the multiple canvas elements are filtered to obtain the target canvas element used as the top-level canvas of the page. This is equivalent to performing a first filtering based on the priority of the canvas elements to obtain the target canvas element. The hierarchical relationship between the target canvas element and its multiple components is determined. Based on the hierarchical relationship, the multiple components are filtered to obtain the target components that can be displayed on the page. This is equivalent to performing a second filtering based on the hierarchical relationship to obtain the target components. Then, the position of the target component in the target canvas element can be determined, and sample labels are constructed based on the position to annotate the corresponding page image, generating training samples for training the component recognition model.

[0312] As can be seen, in the training sample generation method provided in this application embodiment, the canvas elements can be first filtered according to their priority to obtain target canvas elements. Then, based on the hierarchical relationship, multiple components in the target canvas elements are filtered a second time to obtain target components. In other words, for the canvas elements on the page, two different filters can be performed to obtain accurate target components that can be displayed on the page, without requiring the user to manually mark the components on the page. This reduces the time required to determine the target components on the page and improves the efficiency of determining the target components on the page. Furthermore, sample labels can be constructed based on the position of the target components to generate training samples, thereby reducing the time required to generate training samples, improving the efficiency of generating training samples, and improving the accuracy of generating training samples, thus solving technical problem 1.

[0313] After determining the location of the target component, it is possible to further determine whether there is a collision between the second and third components, thereby generating training samples for training the collision recognition model. This can reduce the time required to generate training samples, improve the efficiency of generating training samples, and improve the accuracy of generating training samples, thus solving technical problem 1.

[0314] The following description continues to illustrate the exemplary structure of the training sample generation device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module in the training sample generation device 455 stored in the memory 450 may include:

[0315] The acquisition module 4551 is used to acquire multiple canvas elements in the page, wherein the multiple canvas elements have corresponding priorities and each canvas element includes multiple components.

[0316] The filtering module 4552 is used to filter the plurality of canvas elements based on the priority of each canvas element to obtain the target canvas element to be used as the top-level canvas of the page.

[0317] The determination module 4553 is used to determine the target canvas element and the hierarchical relationship between multiple components in the target canvas element;

[0318] The filtering module 4552 is further configured to filter the multiple components based on the hierarchical relationship to obtain target components that can be displayed on the page;

[0319] The determining module 4553 is further configured to determine the position of the target component in the target canvas element, construct sample labels based on the position, annotate the page image corresponding to the page, and generate training samples for training the component recognition model.

[0320] In some embodiments, the priority of the canvas elements includes inter-layer priority for controlling the rendering order. The filtering module 4552 is further configured to sort the plurality of canvas elements in descending order of the inter-layer priority to obtain a canvas element sequence; obtain the area of ​​the page; and filter the plurality of canvas elements based on the area of ​​the page and the canvas element sequence to obtain a target canvas element to be used as the top-level canvas of the page.

[0321] In some embodiments, the filtering module 4552 is further configured to determine a first area ratio of the canvas elements according to the order of the canvas elements in the canvas element sequence; wherein, the first area ratio is the ratio of the area of ​​the canvas element to the area of ​​the page; and the first canvas element in the canvas element sequence whose first area ratio is greater than a first ratio threshold is determined as the target canvas element.

[0322] In some embodiments, the filtering module 4552 is further configured to determine, based on the hierarchical relationship, components with a hierarchy level lower than a hierarchy threshold among the plurality of components as first components, wherein the number of first components is plurality; and to filter the plurality of first components based on the hierarchy level and area of ​​the first components to obtain target components that can be displayed on the page.

[0323] In some embodiments, the plurality of first components include a plurality of levels. For any one of the first components, the first component at the upper level of the first component is the parent component of the first component, and the first component at the lower level of the first component is the child component of the first component.

[0324] The filtering module 4552 is further configured to, for each first component in the hierarchy, determine the first component and its corresponding sub-components as target components that can be displayed on the page when the first component meets the target conditions; wherein, the target conditions are: when the first component does not have a parent component, the second area ratio of the first component is greater than or equal to the second ratio threshold, wherein the second area ratio is the ratio of the area of ​​the first component to the area of ​​the target canvas element; when the first component has a parent component, the third area ratio of the parent component of the first component is greater than or equal to the second ratio threshold, wherein the third area ratio is the ratio of the area of ​​the parent component of the first component to the area of ​​the target canvas element; when the first component has a parent component, the third area ratio of the parent component of the first component is less than the second ratio threshold, and the second area ratio of the first component is greater than or equal to the second ratio threshold.

[0325] In some embodiments, the plurality of first components include a first level and a second level. The filtering module 4552 is further configured to determine the first component and its sub-components as target components that can be displayed on the page if the second area ratio of the first component in the first level is greater than or equal to a second ratio threshold. The second area ratio is the area ratio of the first component to the target canvas element, and the second level of the sub-component is the next level after the first level.

[0326] If the second area ratio of the first component in the first level is less than the second ratio threshold, and the second area ratio of the first component in the second level is greater than or equal to the second ratio threshold, then the first component in the second level is determined as a target component that can be displayed on the page.

[0327] In some embodiments, the filtering module 4552 is further configured to determine the transparency of the image component after identifying the component with a level lower than the level threshold among the plurality of components as the first component, in the case that the first component is an image component.

[0328] The following conditions are met to trigger the process of filtering multiple first components based on the hierarchy and area of ​​the first component to obtain a target component that can be displayed on the page: the transparency of the image component is greater than a transparency threshold; or the area of ​​the image component is less than a third area threshold.

[0329] In some embodiments, the number of target components is multiple, and the multiple target components include a second component for displaying text; the determining module 4553 is further configured to, after determining the position of the target component in the target canvas element, if the target component includes a third component, determine the collision relationship between the second component and the third component in the page based on the position of the second component and the position of the third component; wherein, the distance between the third component and the second component is less than a third distance threshold; if the collision relationship indicates that there is a collision between the second component and the third component, determine the target position based on the position of the second component and the position of the third component; construct sample labels based on the target position, and annotate the page image corresponding to the page to obtain training samples for training the collision recognition model.

[0330] In some embodiments, the second component and the third component are both rectangular in shape. The position of the second component includes a first angular coordinate and a second angular coordinate, and the position of the third component includes a third angular coordinate and a fourth angular coordinate. The first angular coordinate, the second angular coordinate, the third angular coordinate, and the fourth angular coordinate are in the same two-dimensional coordinate system. The first angular coordinate and the third angular coordinate correspond to each other in the rectangle, and the second angular coordinate and the fourth angular coordinate correspond to each other in the rectangle.

[0331] The determination module 4553 is further configured to determine the collision relationship as follows: if the horizontal coordinate of the first angular coordinate is greater than the horizontal coordinate of the third angular coordinate, and the horizontal coordinate of the second angular coordinate is less than the horizontal coordinate of the fourth angular coordinate, then the collision relationship is determined to be that the second component and the third component collide on the page.

[0332] In some embodiments, the determining module 4553 is further configured to determine the difference between the ordinate of the second angular coordinate and the ordinate of the fourth angular coordinate;

[0333] If the difference is greater than or equal to the difference threshold, the second angular coordinate and the fourth angular coordinate are determined as the coordinates of the target position;

[0334] If the difference is less than the difference threshold, the ordinate of the first angular coordinate and the ordinate of the third angular coordinate are determined as the ordinate of the target position, and the abscissa of the second angular coordinate and the abscissa of the fourth angular coordinate are determined as the abscissa of the target position.

[0335] In some embodiments, the determining module 4553 is further configured to obtain the world coordinates of the target component in the world coordinate system; convert the world coordinates from the world coordinate system to the view coordinate system to obtain the view coordinates; and map the view coordinates to the display area in the page to obtain the position of the target component in the target canvas element.

[0336] In some embodiments, the priority includes inter-layer priority and intra-layer priority for controlling the rendering order; the filtering module 4552 is further configured to determine the candidate canvas element with the highest inter-layer priority from the plurality of canvas elements and count the number of the candidate canvas elements; if the number is greater than 1, the candidate canvas element with the highest intra-layer priority is determined as the target canvas element.

[0337] In some embodiments, the filtering module 4552 is further configured to determine the canvas element with the highest priority among the plurality of canvas elements as the first canvas element; based on the first canvas element and the priority of the canvas elements, filter other canvas elements among the plurality of canvas elements to obtain a second canvas element; and determine the first canvas element and the second canvas element as target canvas elements to be used as the top-level canvas of the page.

[0338] In some embodiments, the filtering module 4552 is further configured to, for the other canvas elements, if there are other canvas elements with a first overlapping area smaller than a first area threshold, and the distance between the first canvas element and the other canvas elements is less than or equal to a first distance threshold, then determine the other canvas element with the highest priority as the second canvas element; wherein, the first overlapping area is the overlapping area between the other canvas element and the first canvas element in the page; if there are other canvas elements with a second overlapping area smaller than the first area threshold, and the distance between the other canvas element and the target area is less than or equal to a second distance threshold, then determine the other canvas element with the highest priority as the second canvas element; wherein, the second overlapping area is the overlapping area between the other canvas element and the target area in the page, and the target area is the sum of the area of ​​the first canvas element in the page and the area of ​​the second canvas element in the page.

[0339] In some embodiments, the hierarchical relationship is represented by a tree structure including multiple nodes, wherein the tree structure has the target canvas element as the root node and the components in the target canvas element as non-root nodes; the filtering module 4552 is further configured to, if the number of target canvas elements is 1, determine the component corresponding to the child node of the root node as a target component that can be displayed on the page; if the number of target canvas elements is multiple, and the overlap area between any two target canvas elements is less than a second area threshold, determine the component corresponding to the child node of the root node as a target component that can be displayed on the page.

[0340] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the training sample generation method described above in this application.

[0341] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the training sample generation method provided in this application. For example, ... Figure 3 The method for generating training samples is shown.

[0342] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0343] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0344] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0345] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0346] The training sample generation method provided in this application embodiment can obtain multiple canvas elements on a page, each with a corresponding priority. Each canvas element includes multiple components. Based on the priority of each canvas element, the multiple canvas elements are filtered to obtain the target canvas element used as the top-level canvas of the page. This is equivalent to performing a first filtering based on the priority of the canvas elements to obtain the target canvas element. The hierarchical relationship between the target canvas element and its multiple components is determined. Based on the hierarchical relationship, the multiple components are filtered to obtain the target components that can be displayed on the page. This is equivalent to performing a second filtering based on the hierarchical relationship to obtain the target components. Then, the position of the target component in the target canvas element can be determined, and sample labels are constructed based on the position to annotate the corresponding page image, generating training samples for training the component recognition model.

[0347] As can be seen, in the training sample generation method provided in this application embodiment, the canvas elements can be first filtered according to their priority to obtain target canvas elements. Then, based on the hierarchical relationship, multiple components in the target canvas elements are filtered a second time to obtain target components. In other words, for the canvas elements on the page, two different filters can be performed to obtain accurate target components that can be displayed on the page, without requiring the user to manually mark the components on the page. This reduces the time required to determine the target components on the page and improves the efficiency of determining the target components on the page. Furthermore, sample labels can be constructed based on the position of the target components to generate training samples, thereby reducing the time required to generate training samples, improving the efficiency of generating training samples, and improving the accuracy of generating training samples, thus solving technical problem 1.

[0348] After determining the location of the target component, it is possible to further determine whether there is a collision between the second and third components, thereby generating training samples for training the collision recognition model. This can reduce the time required to generate training samples, improve the efficiency of generating training samples, and improve the accuracy of generating training samples, thus solving technical problem 1.

[0349] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for generating training samples, characterized in that, The method includes: Get multiple canvas elements in the page, each of which has a corresponding priority, and each canvas element includes multiple components; Based on the priority of each canvas element, the multiple canvas elements are filtered to obtain the target canvas element to be used as the top-level canvas of the page. Determine the target canvas element and the hierarchical relationship between multiple components within the target canvas element; Based on the hierarchical relationship, the multiple components are filtered to obtain the target components that can be displayed on the page; The position of the target component within the target canvas element is determined, and sample labels are constructed based on the position. The page image corresponding to the page is then labeled to generate training samples for training the component recognition model.

2. The method according to claim 1, characterized in that, The priority of the canvas elements includes inter-layer priority for controlling the rendering order; the step of filtering the plurality of canvas elements based on their priorities to obtain the target canvas element to be used as the top-level canvas of the page includes: The multiple canvas elements are sorted in descending order of interlayer priority to obtain a canvas element sequence. The area of ​​the page is obtained, and based on the area of ​​the page and the sequence of canvas elements, the multiple canvas elements are filtered to obtain the target canvas element to be used as the top-level canvas of the page.

3. The method according to claim 2, characterized in that, The step of filtering the plurality of canvas elements based on the area of ​​the page and the sequence of canvas elements to obtain the target canvas element to be used as the top-level canvas of the page includes: According to the order of the canvas elements in the canvas element sequence, determine the first area ratio of the canvas elements respectively; Wherein, the first area ratio is the ratio of the area of ​​the canvas element to the area of ​​the page; The first canvas element in the canvas element sequence whose first area ratio is greater than the first ratio threshold is determined as the target canvas element.

4. The method according to claim 1, characterized in that, The step of filtering the multiple components based on the hierarchical relationship to obtain target components that can be displayed on the page includes: Based on the hierarchical relationship, the component with a level lower than the level threshold among the multiple components is determined as the first component, and the number of the first components is multiple; Based on the hierarchy and area of ​​the first component, multiple first components are filtered to obtain target components that can be displayed on the page.

5. The method according to claim 4, characterized in that, The multiple first components include multiple levels. For any one of the first components, the first component at the upper level of the first component is the parent component of the first component, and the first component at the lower level of the first component is the child component of the first component. The step of filtering multiple first components based on their hierarchy and area to obtain target components that can be displayed on the page includes: For each first component in the aforementioned level, if the first component meets the target condition, then the first component and its corresponding sub-components are determined as target components that can be displayed on the page. The target condition is as follows: When the first component does not have a parent component, the second area ratio of the first component is greater than or equal to the second ratio threshold, wherein the second area ratio is the ratio of the area of ​​the first component to the area of ​​the target canvas element; When the first component has a parent component, the third area ratio of the parent component of the first component is greater than or equal to the second ratio threshold, wherein the third area ratio is the area ratio of the parent component of the first component to the target canvas element. When the first component has a parent component, the third area ratio of the parent component of the first component is less than the second ratio threshold, and the second area ratio of the first component is greater than or equal to the second ratio threshold.

6. The method according to claim 4, characterized in that, The plurality of first components include a first level and a second level. The step of filtering the plurality of first components based on their level and area to obtain target components that can be displayed on the page includes: If the second area ratio of the first component in the first level is greater than or equal to the second ratio threshold, then the first component and its sub-components are determined as target components that can be displayed on the page; wherein, the second area ratio is the area ratio of the first component to the target canvas element, and the second level of the sub-component is the next level below the first level; If the second area ratio of the first component in the first level is less than the second ratio threshold, and the second area ratio of the first component in the second level is greater than or equal to the second ratio threshold, then the first component in the second level is determined as a target component that can be displayed on the page.

7. The method according to claim 4, characterized in that, After determining the component with a level lower than the level threshold among the plurality of components as the first component, the method further includes: If the first component is an image component, determine the transparency of the image component; The following conditions are met to trigger the filtering of multiple first components based on the hierarchy and area of ​​the first component to obtain a target component that can be displayed on the page: The transparency of the image component is greater than the transparency threshold; The area of ​​the image component is less than the third area threshold.

8. The method according to any one of claims 1-7, characterized in that, The number of target components is multiple, and the multiple target components include a second component for displaying text; after determining the position of the target component in the target canvas element, the method further includes: If the target component includes a third component, the collision relationship between the second component and the third component on the page is determined based on the position of the second component and the position of the third component. Wherein, the distance between the third component and the second component is less than a third distance threshold; When the collision relationship characterizes a collision between the second component and the third component, the target position is determined based on the positions of the second component and the third component. Based on the target location, sample labels are constructed, and the page image corresponding to the page is labeled to obtain training samples for training the collision recognition model.

9. The method according to claim 8, characterized in that, Both the second component and the third component are rectangular in shape. The position of the second component includes a first angular coordinate and a second angular coordinate, and the position of the third component includes a third angular coordinate and a fourth angular coordinate. The first angular coordinate, the second angular coordinate, the third angular coordinate, and the fourth angular coordinate are in the same two-dimensional coordinate system. Wherein, the first angular coordinate corresponds to the position of the third angular coordinate in the rectangle, and the second angular coordinate corresponds to the position of the fourth angular coordinate in the rectangle; Determining the collision relationship between the second component and the third component on the page based on the positions of the second component and the third component includes: If the x-coordinate of the first angular coordinate is greater than the x-coordinate of the third angular coordinate, and the x-coordinate of the second angular coordinate is less than the x-coordinate of the fourth angular coordinate, then the collision relationship is determined to be that the second component and the third component collide on the page.

10. The method according to claim 9, characterized in that, Determining the target location based on the position of the second component and the position of the third component includes: Determine the difference between the ordinate of the second angular coordinate and the ordinate of the fourth angular coordinate; If the difference is greater than or equal to the difference threshold, the second angular coordinate and the fourth angular coordinate are determined as the coordinates of the target position; If the difference is less than the difference threshold, the ordinate of the first angular coordinate and the ordinate of the third angular coordinate are determined as the ordinate of the target position, and the abscissa of the second angular coordinate and the abscissa of the fourth angular coordinate are determined as the abscissa of the target position.

11. The method according to any one of claims 1-7, characterized in that, Determining the position of the target component within the target canvas element includes: Obtain the world coordinates of the target component in the world coordinate system; The world coordinates are transformed from the world coordinate system to the view coordinate system to obtain the view coordinates; The view coordinates are mapped to the display area in the page to obtain the position of the target component in the target canvas element.

12. The method according to claim 1, characterized in that, The priority includes inter-layer priority and intra-layer priority for controlling the rendering order; the step of filtering the multiple canvas elements based on the priority of each canvas element to obtain the target canvas element to be used as the top-level canvas of the page includes: From the plurality of canvas elements, determine the candidate canvas element with the highest interlayer priority, and count the number of the candidate canvas elements; If the number is greater than 1, the candidate canvas element with the highest priority within the layer is determined as the target canvas element.

13. The method according to claim 1, characterized in that, The step of filtering the multiple canvas elements based on their priority to obtain the target canvas element to be used as the top-level canvas of the page includes: Among the plurality of canvas elements, the canvas element with the highest priority is determined as the first canvas element; Based on the first canvas element and the priority of the canvas element, other canvas elements among the plurality of canvas elements are filtered to obtain the second canvas element. The first canvas element and the second canvas element are determined as target canvas elements to be used as the top-level canvas of the page.

14. The method according to claim 13, characterized in that, The step of filtering other canvas elements from the plurality of canvas elements based on the first canvas element and the priority of the first canvas element to obtain the second canvas element includes: For the other canvas elements, if there are other canvas elements with a first overlapping area smaller than a first area threshold, and the distance between the first canvas element and the other canvas elements is less than or equal to a first distance threshold, then the other canvas element with the highest priority is determined as the second canvas element. Wherein, the first overlapping area is the overlapping area of ​​the other canvas elements and the first canvas element on the page; If there are other canvas elements with a second overlapping area smaller than the first area threshold, and the distance between the other canvas elements and the target area is less than or equal to the second distance threshold, the other canvas element with the highest priority is determined as the second canvas element. Wherein, the second overlapping area is the overlapping area between the other canvas elements and the target area in the page, and the target area is the sum of the area of ​​the first canvas element in the page and the area of ​​the second canvas element in the page.

15. The method according to claim 1, characterized in that, The hierarchical relationship is represented by a tree structure that includes multiple nodes, with the target canvas element as the root node and the components in the target canvas element as non-root nodes. The step of filtering the multiple components based on the hierarchical relationship to obtain target components that can be displayed on the page includes: If the number of target canvas elements is 1, the component corresponding to the child node of the root node is determined as the target component that can be displayed on the page; If there are multiple target canvas elements, and the overlap area between any two target canvas elements is less than the second area threshold, the component corresponding to the child node of the root node is determined as the target component that can be displayed on the page.

16. A training sample generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire multiple canvas elements in the page. Each of the multiple canvas elements has a corresponding priority, and each canvas element includes multiple components. A filtering module is used to filter the multiple canvas elements based on the priority of each canvas element to obtain the target canvas element to be used as the top-level canvas of the page. The determination module is used to determine the target canvas element and the hierarchical relationship between multiple components in the target canvas element; The filtering module is also used to filter the multiple components based on the hierarchical relationship to obtain target components that can be displayed on the page; The determining module is further configured to determine the position of the target component in the target canvas element, construct sample labels based on the position, annotate the page image corresponding to the page, and generate training samples for training the component recognition model.

17. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the training sample generation method according to any one of claims 1 to 15.

18. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the training sample generation method according to any one of claims 1 to 15.

19. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the training sample generation method according to any one of claims 1 to 15.

Images