Method, device and equipment for constructing target image recognition model training data for transplanting application programs

Abstract

The present invention discloses a method, device and equipment for constructing training data of a target image recognition model for transplanting an application, and relates to application transplantation technology, specifically, taking a screenshot of the transplanted application on a TV terminal, and retrieving the corresponding complete interface, from which the resource data of each operable object in the complete interface is obtained, and the display parameters of the operable object on the TV device are calculated by using the resolution of the current TV and the resolution information of the original terminal of the application, and the contour data of each operable object is obtained by using the obtained display parameters, and the contour marking operation is completed, and then the contour marking results of all operable objects in all interfaces of the transplanting program are summarized to obtain the training set of the image recognition model. The present invention can more quickly and accurately obtain the training data source required for the image recognition model of the transplanted application, and while improving efficiency and accuracy, it can also take into account more models of smart TV devices.

Description

Technical Field

[0001] The present invention relates to the technical field of transplanting application programs, and in particular to a method, device and equipment for constructing target image recognition model training data for transplanting application programs. Background Art

[0002] When porting mobile or cloud applications (APPs, such as cloud games) to smart TVs, in order to enable users to control the APP on the smart TV through the TV remote control, image recognition technology is needed to identify graphics, objects and other operating targets in the application during the porting phase, such as all operable objects in cloud games (function buttons, reward items, system prompts, etc.), which often requires model training based on massive data.

[0003] Generally speaking, in the process of training application image recognition models, people usually use tools to circle the outlines of objects in the pictures, and then export the data required for model training. In the actual implementation of this work, there are many problems:

[0004] 1) Time-consuming. If a game has 40 functional pages and 500 operable objects, it is necessary to manually use a third-party drawing tool to accurately mark each page and each object according to the outline of the object, which consumes a lot of manpower and time.

[0005] 2) Repeatability. Since smart TVs are available in a variety of sizes (common sizes range from 32 inches to 85 inches), and each size has a corresponding resolution, it is necessary to consider the size and display effects of the operating objects of the application ported to the TV at different resolutions, which requires a lot of repetitive work.

[0006] 3) Object overlap. For example, when there is a sliding list in the page of a game app, or when the secondary UI covers the primary UI, the function buttons will overlap, making it difficult to perform reliable outline drawing when processing training data, and thus unable to provide accurate training data to the model; in addition, this problem also puts higher requirements on the data collection of images uploaded by R&D personnel.

[0007] In summary, the preparation of image processing model training data before running the transplanted application on smart TV is currently time-consuming, inefficient, costly, and error-prone. Summary of the invention

[0008] In view of the above, the present invention aims to provide a method, device and equipment for constructing target image recognition model training data for transplanting application programs to solve the specific problems mentioned above.

[0009] The technical solution adopted by the present invention is as follows:

[0010] In a first aspect, the present invention provides a method for constructing target image recognition model training data for transplanting an application, comprising:

[0011] After launching the ported application on the TV device, capture the current screen;

[0012] Determine a corresponding complete scene interface according to the information of the current screen;

[0013] Acquire original resource data of all operable objects in the complete scene interface, the original resource data including the display resolution of the application on the original terminal;

[0014] According to the display resolution and the current resolution of the television device, obtaining the current size information and the current position information of each of the operable objects on the television device;

[0015] Using the current size information and the current position information, obtaining outline data of the operable object on the television device;

[0016] The contour data of all operable objects in all scene interfaces of the application are exported and constructed as a data set for image recognition model training.

[0017] In at least one possible implementation, obtaining the original resource data of all operable objects in the complete scene interface includes: obtaining the original resource data of all operable objects in the complete scene interface from the memory according to the established path information of the complete scene interface.

[0018] In at least one possible implementation manner, the original resource data further includes: original size information and original position information of the operable object.

[0019] In at least one possible implementation manner, a method of obtaining the current size information includes:

[0020] Obtaining a scaling factor using the display resolution and the current resolution;

[0021] The current size information of the operable object on the television device is obtained by combining the scaling factor and the original size information.

[0022] In at least one possible implementation manner, a method of obtaining the current location information includes:

[0023] Acquire current position information of a plurality of first geometric reference points of the operable object;

[0024] The current position information of the first geometric reference point is fused with the current size information, or combined with the original size information of the operable object and the scaling factor to obtain the current position information of several second geometric reference points of the operable object.

[0025] In at least one possible implementation manner, determining the corresponding complete scene interface includes: matching the corresponding complete scene interface through identification information of the current screen.

[0026] In a second aspect, the present invention provides a target image recognition model training data construction device for transplanting an application, comprising:

[0027] A screenshot module, used to capture the current screen after launching the transplanted application on the TV device;

[0028] A complete interface determination module, used to determine the corresponding complete scene interface according to the information of the current screen;

[0029] An object original data acquisition module, used to obtain original resource data of all operable objects in the complete scene interface, wherein the original resource data includes the display resolution of the application on the original terminal;

[0030] A current display information calculation module, used to obtain current size information and current position information of each operable object on the television device according to the display resolution and the current resolution of the television device;

[0031] A current outline data acquisition module, used to obtain outline data of the operable object on the television device using the current size information and the current position information;

[0032] The training set construction module is used to export the contour data of all operable objects in all scene interfaces of the application program and construct the data set for image recognition model training.

[0033] In a third aspect, the present invention provides an electronic device, comprising: one or more processors, a memory and one or more computer programs, wherein the memory may adopt a non-volatile storage medium, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, and when the instructions are executed by the device, the electronic device performs the method as described in the first aspect or any possible implementation manner of the first aspect.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program, which, when executed on a computer, enables the computer to execute the method as described in the first aspect or any possible implementation of the first aspect.

[0035] In a fifth aspect, the present invention further provides a computer program product, which, when executed by a computer, is used to execute the method in the first aspect or any possible implementation of the first aspect. In a possible design of the fifth aspect, the relevant programs involved in the product may be stored in whole or in part on a memory packaged with the processor, or may be stored in whole or in part on a storage medium not packaged with the processor.

[0036] The main design concept of the present invention is to take a screenshot of the transplanted application on the TV, and retrieve the corresponding complete interface, from which the resource data of each operable object in the complete interface is obtained, and the display parameters of the operable objects on the TV device are calculated using the resolution of the current TV and the resolution information of the original terminal of the application, and the contour data of each operable object is obtained using the obtained display parameters to complete the contour marking operation, and then summarize the contour marking results of all operable objects in all interfaces of the transplanted program to obtain the training set of the image recognition model. The present invention can more quickly and accurately obtain the training data source required for the image recognition model of the transplanted application. In actual operation, the present solution can also be packaged in the form of a program package for convenient use, so as to complete the previous work of exporting the training data required for the intelligent image recognition model in simple steps, while improving efficiency and accuracy, and taking into account more intelligent TV device models. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0038] Figure 1 A schematic diagram of a process for constructing target image recognition model training data for transplanting an application provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a target image recognition model training data construction device for transplanting an application provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0041] The present invention proposes an embodiment of a method for constructing target image recognition model training data for transplanting application programs. Specifically, Figure 1 shown, including:

[0042] Step S1, after starting the transplanted application on the TV device, capturing the current picture on the screen;

[0043] Step S2: determining a corresponding complete scene interface according to the information of the current screen;

[0044] To elaborate, the present invention is directed to the development process of porting a certain application from the original device to the TV. Therefore, after a certain application, such as a mobile game, is ported to the TV, the application is started from the TV, and the screen currently displayed on the TV is captured from the program screen after startup. However, those skilled in the art can understand that the screen displayed by the application on the TV is not a complete scene interface. Generally speaking, the operable objects in the complete scene interface also include other content that is not displayed on the current screen (that is, the current screen "outside" also includes other operable objects belonging to the scene interface, and generally operations such as scrolling are required to see these displayed contents). Therefore, the present invention first proposes that the corresponding complete scene interface is automatically detected through the currently captured screen. In actual operation, it can be matched to the "identity" information of the corresponding complete scene interface in the background of the application through, but not limited to, the display text, screen composition, presentation theme and other identifying information of the current screenshot.

[0045] Step S3: obtaining original resource data of all operable objects in the complete scene interface, wherein the original resource data includes the display resolution of the terminal originally carrying the application program;

[0046] By locking the complete scene interface to the operable objects compiled in the program, the problem of requiring a pulley to scroll or overlapping objects that make effective manual marking difficult can be avoided. For example, the screen in the current game can be captured, the corresponding complete interface can be retrieved, and the original resource data of all operable objects in the complete scene interface can be obtained from the memory according to a predetermined path. The original resource data can also include but is not limited to the original size information, original position information, function, type, etc. of the operable objects before transplantation.

[0047] Step S4, obtaining current size information and current position information of each operable object on the television device according to the display resolution and the current resolution of the television device;

[0048] Specifically, the scaling factor can be obtained by first using the display resolution and the current resolution; the current size information of the operable object on the TV device can be obtained by combining the scaling factor and the original size information. As for the position, the current position information of one or more first geometric reference points of the operable object can be further obtained (obtained by using the scaling factor), and then the current position information of the first geometric reference point is fused with the current size information (or combined with the original size information of the operable object and the scaling factor) to obtain the current position information of one or more second geometric reference points of the operable object. It can be understood by those skilled in the art that different geometric reference points can be used according to different operable objects. For example, for a rectangular button, the lower left corner vertex can be used as the first geometric reference point, but is not limited to it, and the position of the upper right corner vertex can be obtained by the above-mentioned processing and calculation.

[0049] Step S5, using the current size information and the current position information, obtaining the outline data of the operable object on the television device;

[0050] Step S6: export the contour data of all operable objects in all scene interfaces of the application program, and construct the contour data for training an image recognition model.

[0051] In summary, the main design concept of the present invention is to take a screenshot of the transplanted application on the TV, and retrieve the corresponding complete interface, from which the resource data of each operable object in the complete interface is obtained, and the display parameters of the operable objects on the TV device are calculated using the resolution of the current TV and the resolution information of the original terminal of the application, and the contour data of each operable object is obtained using the obtained display parameters to complete the contour marking operation, and then summarize the contour marking results of all operable objects in all interfaces of the transplanted program to obtain the training set of the image recognition model. The present invention can more quickly and accurately obtain the training data source required for the image recognition model of the transplanted application. In actual operation, the present solution can also be packaged in the form of a program package for convenient use, so as to complete the previous work of exporting the training data required for the intelligent image recognition model in simple steps, while improving efficiency and accuracy, and taking into account more intelligent TV device models.

[0052] Corresponding to the above-mentioned embodiments and preferred solutions, the present invention also provides an embodiment of a target image recognition model training data construction device 200 for transplanting an application program, such as Figure 2 As shown, it may specifically include the following components:

[0053] A screenshot module 201 is used to capture the current image on the screen after starting the transplanted application on the TV device;

[0054] A complete interface determination module 202, used to determine a corresponding complete scene interface according to the information of the current screen;

[0055] The object original data acquisition module 203 is used to obtain the original resource data of all operable objects in the complete scene interface, and the original resource data includes the display resolution of the application on the original terminal;

[0056] The current display information calculation module 204 is used to obtain the current size information and current position information of each operable object on the television device according to the display resolution and the current resolution of the television device;

[0057] A current outline data acquisition module 205, for obtaining outline data of the operable object on the television device using the current size information and the current position information;

[0058] The training set construction module 206 is used to export the outline data of all operable objects in all scene interfaces of the application program and construct the data set for image recognition model training.

[0059] It should be understood that the above Figure 2 The division of the various components in the target image recognition model training data construction device for transplanting application programs shown is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these components can all be implemented in the form of software calling through processing elements; some components can also be implemented in the form of software calling through processing elements, and some components can be implemented in the form of hardware. For example, one of the above modules can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other components is similar. In addition, all or part of these components can be integrated together, or they can be implemented independently. In the implementation process, each step of the above method or the above components can be completed by an integrated logic circuit of hardware in the processor element or instructions in software form.

[0060] For example, the above components may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, these components may be integrated together to be implemented in the form of a system-on-a-chip (SOC).

[0061] Based on the above embodiments and their preferred solutions, those skilled in the art can understand that, in actual operation, the technical concept involved in the present invention can be applied to a variety of implementations. The present invention uses the following carrier as a schematic illustration:

[0062] (1) An electronic device. The device may specifically include: one or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, and when the instructions are executed by the device, the device performs the steps / functions of the aforementioned embodiment or equivalent implementation.

[0063] The electronic device may specifically be electronic device related to computers, such as but not limited to various computing terminals and electronic products.

[0064] Specifically, the processor, communication interface, and memory can all communicate with each other through a communication bus. Among them, the processor may be a central processing unit CPU, DSP, microcontroller or digital signal processor, and may also include GPU, embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and image signal processor (Image Signal Processor; hereinafter referred to as: ISP). The processor may also include a specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiment of the present invention, etc. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium such as a memory; and the aforementioned memory / storage medium may include: non-volatile memory (Non-Volatile Memory), such as a non-removable disk, U disk, mobile hard disk, optical disk, etc., as well as read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (RandomAccess Memory; hereinafter referred to as: RAM), etc.

[0065] Even though the present invention does not limit the specific form of the electronic device, it can be seen from the introduction of the aforementioned invention that the electronic device here preferably refers to a smart TV or a similar device.

[0066] (2) A computer data storage medium on which a computer program or the above-mentioned device is stored, which, when the computer program or the above-mentioned device is executed, enables a computer to execute the steps / functions of the aforementioned embodiment or equivalent implementation.

[0067] In several embodiments provided by the present invention, any function, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer data storage medium. Based on this understanding, some technical solutions of the present invention are essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of the software product described below.

[0068] It should be particularly pointed out that the storage medium may refer to a server or a similar computer device, specifically, a storage device in a server or a similar computer device storing the aforementioned computer program or the aforementioned apparatus.

[0069] (3) A computer program product (which may include the above-mentioned apparatus), which, when running on a terminal device, enables the terminal device to execute the method for constructing target image recognition model training data for transplanting an application in the above-mentioned embodiment or an equivalent implementation.

[0070] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above implementation method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the above computer program product may include but is not limited to an optimization program library and a program package.

[0071] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c may represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c may be single or multiple.

[0072] Those skilled in the art will appreciate that the various modules, units, and method steps described in the embodiments disclosed in this specification can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0073] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to method embodiments, the relevant parts can refer to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed in multiple places, such as nodes of a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying creative labor.

[0074] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings, but the above is only a preferred embodiment of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched into a variety of equivalent schemes by those skilled in the art without departing from or changing the design ideas and technical effects of the present invention; therefore, the present invention is not limited to the scope of implementation shown in the drawings, and all changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the specification and drawings, should be within the protection scope of the present invention.

Claims

1. A method for constructing target image recognition model training data for transplanting application programs, characterized in that: include: After launching the ported application on the TV device, capture the current screen; Determining a corresponding complete scene interface according to the information of the current picture includes: matching the corresponding complete scene interface through the identification information of the current picture, wherein the complete scene interface includes other content that is not displayed in the captured current picture; Acquire original resource data of all operable objects in the complete scene interface, the original resource data including the display resolution of the application on the original terminal; According to the display resolution and the current resolution of the television device, obtaining the current size information and the current position information of each operable object on the television device, wherein obtaining the current position information of the operable object comprises: obtaining the current position information of a plurality of first geometric reference points of the operable object by using a resolution scaling factor, fusing the current position information of the first geometric reference points with the current size information, and obtaining the current position information of a plurality of second geometric reference points of the operable object; Using the current size information and the current position information, obtaining outline data of the operable object on the television device; The contour data of all operable objects in all scene interfaces of the application are exported and constructed as a data set for image recognition model training.

2. The method for constructing target image recognition model training data for transplanting application programs according to claim 1, characterized in that: The obtaining of original resource data of all operable objects in the complete scene interface includes: obtaining original resource data of all operable objects in the complete scene interface from a memory according to predetermined path information of the complete scene interface.

3. The method for constructing target image recognition model training data for transplanting application programs according to claim 1, characterized in that: The original resource data also includes: original size information and original position information of the operable object.

4. The method for constructing target image recognition model training data for transplanting application programs according to claim 3, characterized in that: Methods for obtaining the current size information include: Obtaining a scaling factor using the display resolution and the current resolution; The current size information of the operable object on the television device is obtained by combining the scaling factor and the original size information.

5. The method for constructing target image recognition model training data for transplanting application programs according to claim 4, characterized in that: The method of obtaining the current location information also includes: The current position information of the first geometric reference point is combined with the original size information of the operable object and the scaling factor to obtain the current position information of a plurality of second geometric reference points of the operable object.

6. A target image recognition model training data construction device for transplanting application programs, characterized in that: include: A screenshot module, used to capture the current screen after launching the transplanted application on the TV device; A complete interface determination module, used to determine a corresponding complete scene interface according to the information of the current picture, including: matching the corresponding complete scene interface through the identification information of the current picture, wherein the complete scene interface includes other content that is not displayed in the captured current picture; An object original data acquisition module, used to obtain original resource data of all operable objects in the complete scene interface, wherein the original resource data includes the display resolution of the application on the original terminal; a current display information calculation module, configured to obtain the current size information and the current position information of each operable object on the television device according to the display resolution and the current resolution of the television device, wherein obtaining the current position information of the operable object comprises: obtaining the current position information of a plurality of first geometric reference points of the operable object by using a resolution scaling factor, fusing the current position information of the first geometric reference points with the current size information, and obtaining the current position information of a plurality of second geometric reference points of the operable object; A current outline data acquisition module, used to obtain outline data of the operable object on the television device using the current size information and the current position information; The training set construction module is used to export the contour data of all operable objects in all scene interfaces of the application program and construct the data set for image recognition model training.

7. An electronic device, characterized in that: include: One or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the target image recognition model training data construction method for transplanting applications as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing target image recognition model training data for transplanting application programs as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for constructing target image recognition model training data for transplanting application programs as described in any one of claims 1 to 5 is implemented.

Images