Method, device, equipment and storage medium for detecting icon blocked by special-shaped screen
The neural network model detects the icon occlusion situation in game video images, solving the problem of low manual detection efficiency in the prior art, and realizing automated and efficient special-shaped screen occlusion icon detection.
Patent Information
- Application Number
- CN202110951394.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-18
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-08-18
AI Technical Summary
In the prior art, the efficiency of detecting the icon blocked by the special-shaped screen in the occlusion is low, and a lot of manpower is required to detect and mark the icon position, resulting in low efficiency.
By obtaining video images of the game screen, using a pre-trained neural network model for occlusion detection, determine whether the icon is blocked by the distinctive area of the special screen, avoiding manual annotation and detection, and improving efficiency.
The efficiency of automatic detection of special-shaped screen occlusion icons is realized, reducing manpower investment, and improving detection efficiency and generalization capabilities.
Smart Images

Figure CN113663327B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of games, and in particular to a method, device, equipment and storage medium for detecting icons blocked by an irregular-shaped screen. Background Art
[0002] With the development of computer technology, the display screens of mobile phones have evolved from ordinary display screens to the current special-shaped screens. However, when using special-shaped screen mobile phones to play mobile games, the special-shaped areas on the special-shaped screen may block the icons on the user interface (UI) in the mobile game.
[0003] In the prior art, in order to avoid the shaped area on the shaped screen from blocking the icons on the UI in the mobile game, the following two methods are usually used to solve the problem: 1) manual detection, by manually observing the game screens on the vast majority of UIs in each mobile game in a large number of mobile phone models, to check whether there is a situation where the icon is blocked by the shaped area on the shaped screen; 2) target detection, based on the automatic detection algorithm of target detection, the position of the icon on the UI in the video image is detected, and after the icon position is detected, it is determined whether the icon is blocked by the shaped area on the shaped screen. However, in the first method mentioned above, whenever new game content is added or modified, a lot of manpower and time are required to detect the occlusion of the icon by the shaped area on the shaped screen; in the second method mentioned above, it is usually necessary to manually mark the position of the icon on the UI in the game screen, and then use algorithms such as target detection to determine the occlusion situation. However, marking the position information of the icon on the UI also requires a lot of manpower and time investment, and it is difficult to have a good generalization ability for any new game screen. Therefore, both of the above two methods have the problem of relying on a large amount of manpower to observe the game screen on the UI and mark the position information of the icons on the UI, which leads to the technical problem of low efficiency in detecting the occlusion of icons by the special-shaped screen.
[0004] To sum up, the existing solutions for detecting the occlusion of icons by irregular-shaped screens have the problem of low efficiency because a large amount of manpower is needed to detect the occlusion of icons by irregular-shaped areas on the irregular-shaped screens. Summary of the invention
[0005] The embodiments of the present invention provide a method, device, equipment and storage medium for detecting icon obstruction by a special-shaped screen, which are used to solve the problem of low efficiency in detecting icon obstruction by a special-shaped area on the special-shaped screen due to the need to rely on a large amount of manpower.
[0006] In a first aspect, an embodiment of the present invention provides a method for detecting an icon blocked by an irregular screen, wherein a terminal device executes a game application and renders a user interface on the irregular screen of the terminal device, wherein the user interface includes a game screen, and the method includes:
[0007] Acquire a video image of a game screen, wherein the video image includes a plurality of picture images, and an edge position of the picture image includes at least one icon;
[0008] The video image is detected based on an occlusion detection model to determine whether the at least one icon is occluded by the irregular area of the irregular screen when displayed on the irregular screen, wherein the occlusion detection model is a model obtained by training a neural network model based on pre-acquired samples and is used to detect whether at least one icon in the image is occluded.
[0009] In a specific embodiment, the method further comprises:
[0010] According to a plurality of sample images including at least one icon acquired in advance, samples for model training are acquired, wherein the samples include positive samples and negative samples, at least one icon in the positive samples is not blocked, and at least one icon in the negative samples is blocked;
[0011] The neural network model is trained according to positive samples and negative samples to obtain an occlusion detection model.
[0012] In a specific implementation, obtaining samples for model training according to a plurality of sample images including at least one icon obtained in advance includes:
[0013] Determine a first mask area, and shield at least one icon in the sample image based on the first mask area;
[0014] A sample image including at least one icon that is not blocked by the first mask area is determined as a positive sample, and a sample image including at least one icon blocked by the first mask area is determined as a negative sample.
[0015] In a specific embodiment, the method further comprises:
[0016] Convert the picture image into a grayscale image, and perform Gaussian blur processing on the grayscale image to obtain a blurred image;
[0017] Performing image binarization processing on the blurred image to determine at least one icon in the blurred image;
[0018] Extracting edge information of at least one icon in the blurred image based on an image edge detection algorithm;
[0019] At least one icon located at an edge position of the picture image is determined based on the edge information.
[0020] In a specific implementation, before detecting the video image based on the occlusion detection model, the method further includes:
[0021] Obtaining the position information of the special-shaped area of the special-shaped screen in the user interface;
[0022] A second mask area is determined based on the special-shaped screen, and the second mask area is set in a position corresponding to the position information in the video image.
[0023] In a specific implementation, detecting a video image based on an occlusion detection model to determine whether at least one icon is occluded by a special-shaped area of the special-shaped screen when displayed on the special-shaped screen includes:
[0024] Determine all frame images of the video image;
[0025] Inputting the frame image into the occlusion detection model for detection to determine whether at least one icon in the frame image is occluded by the second mask area;
[0026] If at least one icon in all frame images is not blocked by the second mask area, it is determined that at least one icon in the video image is not blocked by the special-shaped area.
[0027] In a second aspect, an embodiment of the present invention provides a detection device for an icon blocked by a special-shaped screen, comprising: a processing module and a display module, wherein the processing module executes a game application and renders the game application on the special-shaped screen of the display module to obtain a user interface, wherein the user interface includes a game screen;
[0028] The processing module is further used to obtain a video image of the game screen, wherein the video image includes a plurality of picture images, and the edge position of the picture image includes at least one icon;
[0029] The processing module is also used to detect the video image based on the occlusion detection model to determine whether at least one icon is occluded by the irregular area of the irregular screen when displayed on the irregular screen, wherein the occlusion detection model is a model obtained by training a neural network model based on pre-acquired samples and is used to detect whether at least one icon in the image is occluded.
[0030] In a third aspect, an embodiment of the present invention provides a terminal device, including:
[0031] Processor, memory, display;
[0032] The memory is used to store programs and data, and the processor calls the program stored in the memory to execute the method for detecting an icon blocked by an irregular-shaped screen according to the first aspect.
[0033] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting an icon obstructed by an irregular-shaped screen according to the first aspect.
[0034] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the method for detecting an icon obstructed by an irregular-shaped screen according to the first aspect.
[0035] The method, device, equipment and storage medium for detecting icons blocked by a special-shaped screen provided in an embodiment of the present invention are applied in a terminal device, wherein a game application is executed by the terminal device and a user interface is rendered on the special-shaped screen of the terminal device, wherein the user interface includes a game screen, and a video image of the game screen is obtained, wherein the video image includes a plurality of picture images, and the edge position of the picture image includes at least one icon; the video image is detected based on an occlusion detection model to determine whether at least one icon is blocked by a special-shaped area of the special-shaped screen when displayed on the special-shaped screen, wherein the occlusion detection model is a model obtained by training a neural network model according to pre-acquired samples, and is used to detect whether at least one icon in the image is blocked. In this way, the problem of low efficiency in detecting the blocking of icons by special-shaped areas on the special-shaped screen due to the need to rely on a large amount of manpower to annotate the icons on the user interface can be avoided, thereby greatly improving the efficiency of detecting the blocking of icons by special-shaped areas on the special-shaped screen. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0037] Figure 1 A flowchart of Embodiment 1 of a method for detecting an icon blocked by an irregular-shaped screen provided in an embodiment of the present invention;
[0038] Figure 2 A flowchart of Embodiment 2 of a method for detecting an icon blocked by an irregular-shaped screen provided in an embodiment of the present invention;
[0039] Figure 3 A flowchart of Embodiment 3 of a method for detecting an icon blocked by an irregular-shaped screen provided in an embodiment of the present invention;
[0040] Figure 4 A flowchart of a fourth embodiment of a method for detecting an icon blocked by an irregular-shaped screen provided in an embodiment of the present invention;
[0041] Figure 5 A schematic diagram of the structure of an embodiment of a detection device for an icon blocked by a special-shaped screen provided in an embodiment of the present invention;
[0042] Figure 6 A schematic diagram of the structure of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments made by ordinary technicians in this field under the enlightenment of this embodiment belong to the scope of protection of the present invention.
[0044] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0045] In the several prior arts provided in the background art, there are at least the following technical problems in the scheme for detecting icons blocked by irregular-shaped screens:
[0046] 1) When using the manual detection method, if new game content is added or modified, a lot of manpower and time are required to detect the occlusion of icons by the special-shaped areas on the special-shaped screen, which is inefficient and wastes manpower and time costs.
[0047] 2) When using the target detection method, it is usually necessary to manually mark the position of the icons on the UI in the game screen, and then use algorithms such as target detection to determine the occlusion situation. However, marking the position information of the icons on the UI also requires a lot of manpower and time investment, and it is difficult to have good generalization capabilities for any new game screen.
[0048] In view of the above problems, the present invention proposes a method for detecting icons blocked by a special-shaped screen, which determines the icons located at the edge of the screenshot of the game screen through an image processing method, and then obtains samples for model training based on pre-prepared sample images, and trains a neural network model based on the obtained samples, thereby obtaining an occlusion detection model for detecting the real game video image for testing, so as to determine whether the icons in the game video image are blocked by the special-shaped area on the special-shaped screen, thereby realizing the method of marking the position of the icons in the game screen in a non-manual manner, and improving the efficiency of detecting the occlusion of icons by the special-shaped area on the special-shaped screen. The following first explains the terms involved in the present invention.
[0049] User Interface (UI) is the medium for interaction and information exchange between the system and the user. It realizes the conversion between the internal form of information and the form acceptable to humans.
[0050] Image edge detection algorithm (Edge Detection): The purpose is to significantly reduce the data size of the image while retaining the original image properties. The image edge detection algorithm used in the present invention is the Canny edge detection algorithm.
[0051] Application Programming Interface (API): also known as application programming interface, is an agreement for connecting different components of a software system.
[0052] Software Development Kit (SDK): A collection of development tools used by some software engineers to build application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.
[0053] Terminal device: The terminal device in the present invention refers to a network terminal device used to execute game applications, such as a mobile phone, a tablet computer, etc.
[0054] The core idea of the method for detecting icons obstructed by a special-shaped screen provided by the present invention is to use an image processing method to process a picture image of a screenshot of a game screen, determine the icons located at the edge of the picture image, and then perform different degrees of occlusion processing on these icons to obtain positive samples and negative samples for model training, and use the obtained positive samples and negative samples to train a neural network model to obtain an occlusion detection model, so as to use the occlusion detection model to detect the real game video image to be tested, and determine whether the icon in the video image is obstructed by the special-shaped area on the special-shaped screen. In this way, it is possible to achieve the position of the icon in the game screen without relying on manual methods, thereby greatly improving the efficiency of detecting the occlusion of the icon by the special-shaped area on the special-shaped screen, thereby overcoming the above-mentioned problems that are prone to occur when detecting icons obstructed by special-shaped screens.
[0055] In a specific implementation, the method for detecting icons blocked by a special-shaped screen provided by the embodiment can be applied in an application scenario. In this scenario, the solution is applied in a terminal device installed with a game application, and the game program is used to detect whether an icon at an edge position on the game screen is blocked by a special-shaped area on the special-shaped screen in the game screen. The terminal device can be a smart terminal device such as a mobile phone, a tablet computer, or a computer. The terminal device executes the game application and renders a user interface on the special-shaped screen of the terminal device. The user interface includes the game screen. By using an image processing method to process a picture image of a screenshot of the game screen, the position of the icon can be quickly determined. Then, each frame image of the game video image is detected by using an occlusion detection model obtained by pre-training. If the icons at the edge position in all frame images are not blocked by the special-shaped area on the special-shaped screen, it means that the icons at the edge position in the game video image are not blocked by the special-shaped area on the special-shaped screen. Therefore, the efficiency of detecting the blocking of icons by the special-shaped area on the special-shaped screen can be effectively improved.
[0056] The following describes in detail the method for detecting an icon blocked by a special-shaped screen through several specific implementations.
[0057] Figure 1 The flowchart of the first embodiment of the method for detecting an icon blocked by an irregular screen provided by the embodiment of the present invention is as follows: Figure 1 As shown, the method for detecting an icon blocked by a special-shaped screen includes the following steps:
[0058] S101: Acquire a video image of a game screen.
[0059] In this step, the video image includes multiple picture images, which can be images obtained by taking screenshots of the game screen during the game operation, and there is at least one icon at the edge of the picture image; the video image can be a real game video image obtained during the game operation, which can be obtained through an automated tool such as AirTest, and the video image can also be called a game test video. Since the picture image obtained by the screenshot of the terminal device with a special-shaped screen must be a complete image regardless of whether there is a problem of the special-shaped area on the special-shaped screen blocking the icon, at least one icon on the above picture image is also an unobstructed and complete icon.
[0060] In this solution, the position of the irregular area on the irregular screen is generally a position set in advance by major manufacturers. The irregular area is usually located at the edge of two narrow sides of the user interface displayed on the irregular screen of the terminal device. Therefore, the icon blocked by the irregular area on the irregular screen is also the icon located at the narrow edge of the image in the game screen.
[0061] In the above scheme, since the purpose of this scheme is to detect the problem of icon obstruction by the irregular-shaped area on the irregular-shaped screen, the acquired video image includes at least one icon. If there is no icon in the video image, there is no need to detect the icon obstruction by the irregular-shaped area in the video image.
[0062] S102: Detect the video image based on the occlusion detection model to determine whether at least one icon is occluded by a special-shaped area of the special-shaped screen when displayed on the special-shaped screen.
[0063] In this step, the occlusion detection model is a model obtained by training a neural network model based on pre-acquired samples and used to detect whether at least one icon in an image is occluded. The neural network model can be a classification neural network model DenseNet.
[0064] The method for detecting icons obstructed by a special-shaped screen provided in the present embodiment is by acquiring a video image of a game screen, wherein the video image includes a plurality of picture images, and the edge position of the picture image includes at least one icon; detecting the video image based on an occlusion detection model to determine whether at least one icon is obstructed by a special-shaped area of the special-shaped screen when displayed on the special-shaped screen, wherein the occlusion detection model is a model obtained by training a neural network model based on pre-acquired samples and used to detect whether at least one icon in the image is obstructed. In this way, the problem of low efficiency in detecting the obstruction of icons by special-shaped areas on the special-shaped screen caused by relying on a large amount of manpower to label icons on the user interface can be avoided, thereby improving the efficiency in detecting the obstruction of icons by special-shaped areas on the special-shaped screen.
[0065] The following is a detailed description of the method for training the occlusion detection model.
[0066] Figure 2 The flowchart of the second embodiment of the method for detecting an icon blocked by an irregular screen provided by the embodiment of the present invention is as follows: Figure 2 As shown, the specific implementation process of the detection method of the special-shaped screen blocking icon is as follows:
[0067] S201: Acquire samples for model training based on a plurality of sample images including at least one icon acquired in advance.
[0068] In this step, while the game is running, you can take a screenshot of the game screen to obtain multiple sample images including at least one icon. The samples used for model training include positive samples and negative samples, wherein at least one icon in the positive sample is not obscured and at least one icon in the negative sample is obscured.
[0069] In this scheme, since the sample image obtained by the screenshot of the terminal device equipped with a special-shaped screen must be a complete image regardless of whether there is a problem of the special-shaped area on the special-shaped screen blocking the icon, it is necessary to use a self-supervised learning method to artificially create positive and negative samples based on the sample image, so that the model can be trained using positive and negative samples.
[0070] S202: Train the neural network model according to the positive samples and the negative samples to obtain an occlusion detection model.
[0071] In this step, the neural network model can be a classification neural network model DenseNet. After obtaining the positive samples and the negative samples, the obtained positive samples and negative samples can be input into DenseNet for classification training. Since at least one icon in the positive sample is not blocked and at least one icon in the negative sample is blocked, the occlusion detection model trained in this way can well classify at least one icon in each frame image in the video image, and determine whether at least one icon is blocked, so that the occlusion detection model can be used to detect whether at least one icon in the video image is blocked by the irregular area on the irregular screen.
[0072] In this scheme, during the training of the neural network model, the loss function may use cross-entropy until the neural network begins to converge, and a trained occlusion detection model can be obtained.
[0073] In a specific implementation, samples for model training are obtained based on a plurality of pre-acquired sample images including at least one icon, including: determining a first mask area, and blocking at least one icon in the sample image based on the first mask area; determining the sample image including at least one icon not blocked by the first mask area as a positive sample, and determining the sample image including at least one icon blocked by the first mask area as a negative sample.
[0074] In this scheme, since the sample image is a complete image with no content obscured, in order to simulate the situation where the icon in the sample image is obscured by the irregular area on the irregular screen, when obtaining positive samples and negative samples, the first mask area determined based on the mask can be used to completely or partially obstruct at least one icon in the sample image, so that the sample image including at least one icon not obscured by the first mask area can be determined as a positive sample, and the sample image including at least one icon obscured by the first mask area can be determined as a negative sample.
[0075] In the above scheme, in the specific implementation, a first mask area can be set, and then the first mask area can be used to partially or completely block at least one icon in the sample image. The size of the first mask area can be changed according to actual needs, so that a large number of positive samples and negative samples can be batch produced.
[0076] In the above scheme, since the use of the mask can be considered as changing the pixel value of part of the area or the entire icon area on the icon to 0, that is, the icon blocked by the mask is black, the mask can be used to block the icon on the sample image to obtain positive samples and negative samples.
[0077] The following is a detailed description of a method for determining at least one icon at an edge position of a picture image.
[0078] Figure 3 The flowchart of the third embodiment of the method for detecting an icon blocked by an irregular screen provided by the embodiment of the present invention is as follows: Figure 3 As shown, the specific implementation process of the detection method of the special-shaped screen blocking icon is as follows:
[0079] S301: Convert the picture image into a grayscale image, and perform Gaussian blur processing on the grayscale image to obtain a blurred image.
[0080] In this step, the picture image includes icon content and other content such as characters, scenes, etc. Generally speaking, icons usually have relatively obvious frames. For example, rectangular icons have obvious rectangular frames (such as exit buttons, etc.), circular icons have obvious circular frames (such as skill buttons, etc.), etc., while other content such as characters, scenes, etc. usually do not have relatively obvious frames compared to icons. Therefore, the color picture image can be converted into a grayscale image, and the grayscale image can be Gaussian blurred according to the texture features of the Gaussian blurred smoothed image. In the resulting blurred image, the edge information of the icon frame can be retained to a certain extent, while the texture features of other content such as characters, scenes, etc. will be blurred.
[0081] S302: performing image binarization processing on the blurred image to determine at least one icon in the blurred image.
[0082] In this step, after obtaining the blurred image, since the icons in the blurred image have more obvious frames compared to other contents such as characters, scenes, etc., and in this solution, it is necessary to determine at least one icon in the blurred image, therefore, the blurred image can be binarized to achieve the determination of the more obvious icon.
[0083] S303: Extract edge information of at least one icon in the blurred image based on an image edge detection algorithm.
[0084] In this step, since it is necessary to determine the icons at the edge position of the picture image, after determining at least one icon in the blurred image, it is necessary to extract the edge information of at least one icon in the blurred image through an image edge detection algorithm, and determine the position of at least one icon on the blurred image through this edge information.
[0085] S304: Determine at least one icon located at an edge position of the picture image based on the edge information.
[0086] In this step, the distance between at least one icon and the narrow edge of the image can be determined from the edge information of at least one icon extracted by the image edge detection algorithm. When judging whether at least one icon is located at the edge of the image, a distance threshold can be set in advance. When the distance between the icon and the narrow edge of the image is greater than the above distance threshold, it can be determined that the icon is not located at the narrow edge of the image. When the distance between the icon and the narrow edge of the image is less than the above distance threshold, it can be determined that the icon is located at the narrow edge of the image. Thus, the icon located at the narrow edge of the image is detected to determine whether it is blocked.
[0087] The method for detecting icons obstructed by irregular-shaped screens provided in the present embodiment can determine the position information of at least one icon on the image by performing image processing on the picture image, thereby avoiding the problem of inefficiency caused by manually marking the position information of icons, thereby solving a large amount of manpower and time costs, and improving the efficiency of detecting the obstruction of icons by irregular-shaped areas on the irregular-shaped screen.
[0088] The following is a detailed description of the method for detecting video images using an occlusion detection model.
[0089] Figure 4 This is a flow chart of Embodiment 4 of the method for detecting an icon blocked by an irregular screen provided in an embodiment of the present invention, as shown in FIG. Figure 4 As shown, the specific implementation process of the detection method of the special-shaped screen blocking icon is as follows:
[0090] S401: Acquire position information of a special-shaped area of the special-shaped screen in the user interface.
[0091] In this step, the position information of the irregular area on the irregular screen of different terminal devices in the user interface can be obtained through the API or SDK provided by major manufacturers. For example, when the terminal device is a mobile phone, the position of the irregular area is the position of the upper narrow edge of the mobile phone screen.
[0092] S402: Determine a second mask area based on the irregular-shaped area, and set the second mask area in a position corresponding to the position information in the video image.
[0093] In this step, since the video image is a real game video obtained during the game running process, the video image is also a complete image with no content blocked. Therefore, when detecting the video image, the alien area can be set in the corresponding position in the video image in the form of a mask according to the original position of the alien area on the user interface of the terminal device, so as to achieve the purpose of using a mask to simulate the alien area to block at least one icon in the video image.
[0094] In the above scheme, due to different terminal devices, the size and shape of the shaped area of the shaped screen may be different. For example, for a mobile phone with an iOS operating system, the shape of the shaped area of the shaped screen is a rounded rectangle, which is commonly known as bangs by users, and the area of the shaped area of the shaped screen of the iOS system is larger, while for a mobile phone with an Android operating system, the shape of the shaped area of the shaped screen may be a water drop shape or a circle, and the area of the shaped area of the shaped screen is smaller. Therefore, a second mask area can be determined according to the size of the shaped area of the shaped screen, and then the second mask area can be set in a position in the video image corresponding to the position information of the shaped area in the user interface.
[0095] In a specific implementation, a video image is detected based on an occlusion detection model to determine whether at least one icon is occluded by a shaped area of the shaped screen when displayed on the shaped screen, including: determining all frame images of the video image; inputting the frame image into the occlusion detection model for detection to determine whether at least one icon in the frame image is occluded by a second mask area; if at least one icon in all frame images is not occluded by the second mask area, then determining that at least one icon in the video image is not occluded by the shaped area.
[0096] In this scheme, since the occlusion detection model can detect picture images, the acquired video image can be divided into multiple frame images, and then each frame image is input into the occlusion detection model for detection. If the icons located at the narrow edge position on all frame images are not blocked by the second mask area, it can be determined that the icons located at the narrow edge position in the video image are not blocked by the irregular area of the irregular screen. Correspondingly, if in at least one frame image, at least one icon located at the narrow edge is completely or partially blocked by the second mask area, it can be determined that the icon located at the narrow edge position in the video image is blocked by the irregular area of the irregular screen.
[0097] The method for detecting icons blocked by a special-shaped screen of the present invention obtains a video image of a game screen, wherein the video image includes a plurality of picture images, and the edge position of the picture image includes at least one icon; the video image is detected based on an occlusion detection model to determine whether at least one icon is blocked by a special-shaped area of the special-shaped screen when displayed on the special-shaped screen, wherein the occlusion detection model is a model obtained by training a neural network model according to pre-acquired samples, and is used to detect whether at least one icon in the image is blocked. In this way, the problem of relying on a large amount of manpower to annotate icons on the user interface can be avoided, and the method also has strong generalization for new content brought by the new version of the game. At the same time, in actual testing and use, the trained occlusion detection model can replace manual observation and annotation of the position of the icon, and realize automatic completion of the detection of the occlusion of the icon on each frame of the video image, thereby saving a lot of manpower and time costs, and further improving the efficiency of detecting the occlusion of the icon by the special-shaped area of the special-shaped screen.
[0098] Generally speaking, the technical solution provided by the present invention automatically marks the positions of icons on the image through self-supervised learning, which greatly improves the efficiency of obtaining samples, thereby also improving the efficiency of training the neural network model. Then, the game test video is detected according to the occlusion detection model obtained by training the neural network model, which will further improve the efficiency of detecting the occlusion of icons by the irregular areas of the irregular screen. It is a technical implementation method that has both strong generalization and high detection efficiency.
[0099] Figure 5 A schematic diagram of a structure of an embodiment of a detection device for an icon blocked by a special-shaped screen provided in an embodiment of the present invention, as shown in FIG. Figure 5 As shown, the detection device 50 for detecting an icon blocked by a special-shaped screen includes:
[0100] A processing module 51 and a display module 52, wherein the processing module 51 executes the game application and renders the game application on the special-shaped screen of the display module 52 to obtain a user interface, wherein the user interface includes a game screen;
[0101] The processing module 51 is further used to obtain a video image of the game screen, wherein the video image includes a plurality of picture images, and the edge position of the picture image includes at least one icon;
[0102] The processing module 51 is also used to detect the video image based on the occlusion detection model to determine whether at least one icon is occluded by the irregular area of the irregular screen when displayed on the irregular screen, wherein the occlusion detection model is a model obtained by training a neural network model based on pre-acquired samples and is used to detect whether at least one icon in the image is occluded.
[0103] Optionally, the processing module 51 is also used to obtain samples for model training based on a plurality of sample images including at least one icon acquired in advance, the samples including positive samples and negative samples, at least one icon in the positive samples is not occluded, and at least one icon in the negative samples is occluded; the neural network model is trained based on the positive samples and the negative samples to obtain an occlusion detection model.
[0104] Optionally, the processing module 51 is also used to determine the first mask area, based on which the first mask area blocks at least one icon in the sample image; determine the sample image including at least one icon not blocked by the first mask area as a positive sample, and determine the sample image including at least one icon blocked by the first mask area as a negative sample.
[0105] Optionally, the processing module 51 is also used to convert the picture image into a grayscale image, and perform Gaussian blur processing on the grayscale image to obtain a blurred image; perform image binarization processing on the blurred image to determine at least one icon in the blurred image; extract edge information of at least one icon in the blurred image based on an image edge detection algorithm; and determine at least one icon located at the edge position of the picture image based on the edge information.
[0106] Optionally, the processing module 51 is further configured to obtain position information of the irregular region of the irregular screen in the user interface; determine a second mask region based on the irregular screen, and set the second mask region at a position in the video image corresponding to the position information.
[0107] Optionally, the processing module 51 is also used to determine all frame images of the video image; input the frame image into the occlusion detection model for detection to determine whether at least one icon in the frame image is occluded by the second mask area; if at least one icon in all frame images is not occluded by the second mask area, it is determined that at least one icon in the video image is not occluded by the special-shaped area.
[0108] The device for detecting an icon blocked by a special-shaped screen provided in this embodiment is used to execute the technical solution in the aforementioned method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.
[0109] Figure 6 A schematic diagram of the structure of a terminal device provided in an embodiment of the present invention is shown in FIG. Figure 6 As shown, the terminal device 600 includes:
[0110] Processor 611, memory 612, display 613;
[0111] The memory 612 is used to store programs and data, and the processor 611 calls the program stored in the memory to execute the technical solution of the method for detecting an icon blocked by an irregular-shaped screen provided in the embodiment shown in the above method.
[0112] In the above terminal device, the memory 612 and the processor 611 are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines, such as through a bus connection. The memory 612 stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the memory in the form of software or firmware. The processor 611 executes various functional applications and data processing by running the software programs and modules stored in the memory 612.
[0113] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory is used to store programs, and the processor executes the programs after receiving the execution instructions. Furthermore, the software programs and modules in the above-mentioned memory may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.
[0114] The processor may be an integrated circuit chip having the ability to process signals. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0115] The embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium includes a program, and when the program is executed by a processor, it is used to implement the technical solution of the method for detecting an icon blocked by an irregular-shaped screen provided in the method embodiment.
[0116] The present application also provides a computer program product, including: a computer program, which, when executed by a processor, is used to implement the technical solution of the method for detecting an icon blocked by an irregular-shaped screen provided in the aforementioned method embodiment.
[0117] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for detecting icons blocked by special-shaped screens, It is characterized in that The game application is executed by a terminal device and rendered on a special-shaped screen of the terminal device to obtain a user interface, wherein the user interface includes a game screen, and the method includes: Acquire a video image of the game screen, wherein the video image includes a plurality of picture images, and an edge position of the picture image includes at least one icon; Acquire position information of the irregular-shaped area of the irregular-shaped screen in the user interface; Determine a second mask area based on the irregular-shaped area, and set the second mask area in a position corresponding to the position information in the video image; Detecting the video image with the second mask area set based on the occlusion detection model to determine whether the at least one icon is occluded by the irregular area of the irregular screen when displayed on the irregular screen, wherein the occlusion detection model is a model obtained by training a neural network model based on pre-acquired samples and used to detect whether the at least one icon in the image is occluded, and the samples are obtained based on a self-supervised learning method; The method further comprises: Convert the picture image into a grayscale image, and perform Gaussian blur processing on the grayscale image to obtain a blurred image; Performing image binarization processing on the blurred image to determine the at least one icon in the blurred image; Extracting edge information of the at least one icon in the blurred image based on an image edge detection algorithm; the edge information includes a distance between the at least one icon and a narrow edge of the blurred image; When the distance between the icon and the narrow edge of the blurred image is less than a distance threshold, the icon is located at the narrow edge of the blurred image, and the icon located at the narrow edge of the image is detected using an occlusion detection model.
2. The method according to claim 1, It is characterized in that The method further comprises: Acquire samples for model training according to a plurality of sample images including the at least one icon acquired in advance, wherein the samples include positive samples and negative samples, wherein the at least one icon in the positive samples is not blocked, and the at least one icon in the negative samples is blocked; The neural network model is trained according to the positive samples and the negative samples to obtain the occlusion detection model.
3. The method according to claim 2, It is characterized in that The step of acquiring samples for model training according to a plurality of sample images including the at least one icon acquired in advance comprises: Determine a first mask area, and shield at least one of the icons in the sample image based on the first mask area; A sample image including the at least one icon that is not blocked by the first mask area is determined as the positive sample, and a sample image including the at least one icon blocked by the first mask area is determined as the negative sample.
4. The method according to claim 1, It is characterized in that The detecting the video image based on the occlusion detection model to determine whether the at least one icon is occluded by the irregular region of the irregular screen when displayed on the irregular screen includes: Determine all frame images of the video image; Inputting the frame image into the occlusion detection model for detection to determine whether the at least one icon in the frame image is occluded by the second mask area; If the at least one icon in all the frame images is not blocked by the second mask area, it is determined that the at least one icon in the video image is not blocked by the special-shaped area.
5. A detection device for icons blocked by special-shaped screens, It is characterized in that include: A processing module and a display module, wherein the processing module executes the game application and renders the game application on the special-shaped screen of the display module to obtain a user interface, wherein the user interface includes a game screen; The processing module is further used to obtain a video image of the game screen, wherein the video image includes a plurality of picture images, and the edge position of the picture image includes at least one icon; The processing module is further used to obtain position information of the irregular region of the irregular screen in the user interface; determine a second mask region based on the irregular region, and set the second mask region in a position corresponding to the position information in the video image; The processing module is further used to detect the video image with the second mask area set based on the occlusion detection model to determine whether the at least one icon is occluded by the irregular area of the irregular screen when displayed on the irregular screen, wherein the occlusion detection model is a model obtained by training a neural network model based on pre-acquired samples and used to detect whether the at least one icon in the image is occluded, and the samples are obtained based on a self-supervised learning method; The processing module is further used to convert the picture image into a grayscale image, and perform Gaussian blur processing on the grayscale image to obtain a blurred image; Performing image binarization processing on the blurred image to determine the at least one icon in the blurred image; Extracting edge information of the at least one icon in the blurred image based on an image edge detection algorithm; the edge information includes a distance between the at least one icon and a narrow edge of the blurred image; When the distance between the icon and the narrow edge of the blurred image is less than a distance threshold, the icon is located at the narrow edge of the blurred image, and the icon located at the narrow edge of the image is detected using an occlusion detection model.
6. A terminal device, It is characterized in that include: Processor, memory, display; The memory is used to store programs and data, and the processor calls the program stored in the memory to execute the method for detecting an icon blocked by an irregular-shaped screen according to any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for detecting an icon blocked by an irregular-shaped screen as described in any one of claims 1 to 4 is implemented.
8. A computer program product, It is characterized in that It comprises a computer program, which is used to implement the method for detecting an icon blocked by an irregular-shaped screen as described in any one of claims 1 to 4 when executed by a processor.
Citation Information
Patent Citations
Electronic device, game interface display method and related product
CN108170346A
Two-dimensional code location and detection method
CN108985129A
Shielded target detection method, system and device and storage medium
CN110222764A
Image detection method and device and computer readable storage medium
CN111310613A
