A multi-type screen positioning method in visual images

Abstract

This application relates to a method for locating multiple types of screens in a visual image. The method includes: identifying images S using the Mask R-CNN instance segmentation algorithm. input The device is located in a specific area. This area is processed to ensure that the image of the area contains the entire device screen. Finally, an image S containing the entire device screen is cropped out. rough For image S rough Multiple candidate screen contours are identified through edge detection. Based on the screen shape features, a heuristic method is used to fit polygons to all candidate contours. The candidate contours are then sorted in descending order of their fit with the fitted polygons. Based on the sorted candidate contours, the shape category of the device screen is determined, and different localization algorithms are used to locate the screen region. The coordinates of the four vertices of the screen are obtained, and the quadrilateral region formed by these four vertices is used as the final identified and located device screen region. This improves the accuracy of identifying the screen region in an image.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a method for multi-type screen positioning in visual images. Background Technology

[0002] With the rapid development of the electronics manufacturing industry, devices with electronic displays have become a trend across various industries. In some fields, identifying and extracting precise areas of a device's screen from images or videos has become an indispensable part of implementing related technologies. For example, in screen content extraction, it's necessary to accurately locate the screen area in an image and remove the background before proceeding to the next step of content extraction. Similarly, in robot-based visual testing, the testing robot needs to locate the precise screen area of ​​the test device to further test the user interface of the application within the device. The accuracy of screen area localization directly affects the accuracy of subsequent work.

[0003] Accurately locating screen regions in images presents numerous challenges. With technological advancements, screen types have diversified, including right-angled screens, rounded-corner screens, and notch screens, further complicating this task. Several approaches to screen region localization exist: 1. Color-based methods: This involves converting the input image to a color space, identifying regions that match the target color characteristics, and then performing recognition operations. This method considers only limited color information, often failing to effectively locate screen regions when the screen and surrounding environment share similar colors. 2. Edge-based methods: These methods locate the screen by analyzing the differences in grayscale or structural information between the target region and other regions. However, this method often fails to identify the true edges of the screen region, and is difficult to use effectively when some edge information is missing. 3. Basic object detection and instance segmentation algorithms, such as YOLO and Mask R-CNN. These algorithms can only identify the approximate location of the device screen in an image. The resulting image may also contain other environmental objects outside the screen, significantly interfering with subsequent work such as screen content extraction and visualization testing. In summary, the above three methods cannot handle some special cases, such as missing information at some screen edges or when the rectangular image being displayed on the screen is aligned with the screen edge. Therefore, the accuracy of identifying the screen area in an image is currently low. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for locating multiple screen types in visual images that can improve the accuracy of identifying the screen area in an image, addressing the aforementioned technical problems.

[0005] A method for locating multiple screen types in a visual image, used to identify a device screen in an image and locate its location, the method comprising:

[0006] Step S1: Use the instance segmentation algorithm of Mask R-CNN to identify image S input The device in the diagram is located in region M.

[0007] Step S2: Process the region M so that the image of region M contains the entire device screen, and crop out the image S containing the entire device screen. rough ;

[0008] Step S3: Process the image S rough Multiple candidate screen contours are identified through edge detection. Based on the screen shape features, a heuristic method is used to fit polygons to all candidate contours. The candidate contours are sorted in descending order of their fit with the fitted polygons.

[0009] Step S4: Based on the sorted candidate contours, determine the shape category of the device screen, use different positioning algorithms to locate the screen area, obtain the coordinates of the four vertices of the screen, and use the quadrilateral area formed by the four vertices as the final identified and located device screen area.

[0010] The aforementioned multi-type screen localization method for visual images identifies image S by utilizing the instance segmentation algorithm of Mask R-CNN. input In the context of the device, the region M where the device is located is determined. Region M is processed so that its image encompasses the entire device screen. An image S containing the entire device screen is then cropped. rough For the image S rough Multiple candidate screen contours are identified through edge detection. Based on the screen shape features, a heuristic method is used to fit polygons to all candidate contours. The candidate contours are then sorted in descending order of their fit with the fitted polygons. Based on the sorted candidate contours, the shape category of the device screen is determined, and different localization algorithms are used to locate the screen region. The coordinates of the four vertices of the screen are obtained, and the quadrilateral region formed by these four vertices is used as the final identified and located device screen region. This achieves accurate localization of the device screen region from the image, improving the accuracy of identifying the screen area in an image. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a method for locating multiple types of screens in a visual image in one embodiment.

[0012] Figure 2This is a schematic diagram illustrating how the Mask R-CNN algorithm acquires the screen area in this embodiment;

[0013] Figure 3 This is a schematic diagram of the area of ​​the screen under test when M is empty in the embodiment;

[0014] Figure 4 This is a schematic diagram of image preprocessing and candidate contour acquisition in the embodiment;

[0015] Figure 5 This is a schematic diagram comparing the two contours and the fitted polygons in the embodiment;

[0016] Figure 6 This is a schematic diagram illustrating the positioning of the notch on the screen in the embodiment;

[0017] Figure 7 This is a schematic diagram illustrating a special case of right-angle screen positioning in the embodiment;

[0018] Figure 8 This is a schematic diagram of the precise positioning results of the right-angle screen in the embodiment;

[0019] Figure 9 This is a schematic diagram of screen positioning where some screen edge information is missing in the embodiment;

[0020] Figure 10 This is a schematic diagram of the non-right-angle screen positioning process in the embodiment;

[0021] Figure 11 The quadrilateral frame B with the highest fit in the embodiment is... 1st Candidate box diagram;

[0022] Figure 12 This is a schematic diagram of the results of non-right-angle screen positioning in the embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] In one embodiment, such as Figure 1 As shown, a method for locating multiple types of screens in a visual image is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0025] Step S1: Use the instance segmentation algorithm of Mask R-CNN to identify image S input The device in the system is used to determine the region M in which the device is located.

[0026] First, a training dataset is constructed, where each sample is an image of a mobile phone, tablet, laptop, desktop monitor, and in-vehicle display. Each image contains the screen of a single device in an powered-on state, and the label is the device type and the mask of the area where the device is located in each image. Next, an instance segmentation model is trained using the Mask R-CNN instance segmentation algorithm. Then, given an image S containing the complete device and its screen... input The trained instance segmentation model is used to identify image S. input The device in the image S, and find the device in the image S. input The region M in which it is located.

[0027] Step S2: Process the region M so that the image of region M contains the entire device screen, and crop out the image S containing the entire device screen. rough .

[0028] It should be understood that region M is processed to make its image encompass the entire device screen as much as possible.

[0029] Step S3: Process the image S rough Multiple candidate screen contours are identified through edge detection. Based on the screen shape features, a heuristic method is used to fit polygons to all candidate contours. The candidate contours are then sorted in descending order of their fit with the fitted polygons.

[0030] Step S4: Based on the sorted candidate contours, determine the shape category of the device screen, use different positioning algorithms to locate the screen area, obtain the coordinates of the four vertices of the screen, and use the quadrilateral area formed by the four vertices as the final identified and located device screen area.

[0031] First, the shape of the device screen is determined, including right-angled screens, rounded-corner screens, notch screens, and non-right-angled screens such as those with missing edge information due to reflections during shooting. Based on this, different algorithms are used to locate the screen area and obtain the coordinates of the four vertices of the screen. The quadrilateral area formed by the four vertices is the finally identified and located device screen area.

[0032] The aforementioned multi-type screen localization method for visual images identifies image S by utilizing the instance segmentation algorithm of Mask R-CNN. input In the context of the device, the region M where the device is located is determined. Region M is processed so that its image encompasses the entire device screen. An image S containing the entire device screen is then cropped. rough For the image S roughMultiple candidate screen contours are identified through edge detection. Based on the screen shape features, a heuristic method is used to fit polygons to all candidate contours. The candidate contours are then sorted in descending order of their fit with the fitted polygons. Based on the sorted candidate contours, the shape category of the device screen is determined, and different localization algorithms are used to locate the screen region. The coordinates of the four vertices of the screen are obtained, and the quadrilateral region formed by these four vertices is used as the final identified and located device screen region. This achieves accurate localization of the device screen region from the image, improving the accuracy of identifying the screen area in an image.

[0033] In one embodiment, step S2 includes:

[0034] If the region M where the device is located, as determined in step S1, is not empty, determine the minimum bounding rectangle R of the region M. M ;

[0035] The minimum bounding rectangle R of the region M. M Expanding outward by a factor of N1, we obtain the image S input By cropping out environmental interference around the device screen, an image S containing the entire device screen is obtained. rough ;

[0036] If the area M where the device is located, as determined in step S1, is empty, based on the characteristic that the device screen is centered in the captured image, according to the preset image S with width τ×w and height τ×h... input The central region is cropped out of the image S. input Environmental interference around the outer edge of the device screen is analyzed to obtain an image S containing the entire device screen. rough Where w and h are the images S and S, respectively. input The width and height are given, and τ is the cutting ratio parameter, τ = 0.15.

[0037] It should be understood that if the region M found in step S1 is not empty, then the minimum bounding rectangle R of that region should be found. M To prevent R M The device screen is not fully covered, so R M Expanding outward by a factor of N1 (N1 = 0.15), we obtain the original image S. input Cropping out environmental interference around the device screen and including the entire device screen as much as possible in the image. rough If the region M found in step S1 is empty, based on the characteristic that the device screen is generally located in the center of the captured image, according to the preset width τ×w and height τ×h S input The central region (w and h are respectively the image S) input The width and height (τ = 0.15) of the image S are cropped. inputTo minimize environmental interference around the device screen, an image S containing as much of the device screen as possible is obtained. rough .

[0038] In one embodiment, step S3 includes:

[0039] Analyze the image S rough If the resolution is greater than 1920×1080, then the image S... rough Compressed to the image S rough The image is obtained by multiplying the value by N². If the image size is not greater than 1920×1080, no further compression is needed. Then, the compressed image or the image S is further processed. rough Binarization, Gaussian filtering, and opening / closing operations are performed to obtain the preprocessed image S. preprocess Where N2 = 0.25;

[0040] For the preprocessed image S preprocess The Canny edge detection algorithm is used to detect all edges in the image, and the edges are connected by a closing operation to obtain image S. Canny ;

[0041] Image S was extracted using a contour extraction algorithm. Canny The set of all contours in the set C = {C1,...,C2} n}, where C1 is the first contour, C n Let n be the nth contour, where n is the total number of contours;

[0042] Statistical analysis of each contour C i The contour C is calculated by the number of pixels contained within it. i Given the area of ​​the contour, extract the contour with the largest area from set C and use it as the first candidate contour C for the screen. a Add to candidate contour set C candidate In the context, i = 1, ..., n;

[0043] In candidate contour C a Three candidate contours are selected sequentially from the sub-contours, where the selection method is as follows: In candidate contour C... a Select the candidate contour C with the largest area from the sub-contours. b In candidate contour C b Select the candidate contour C with the largest area from the sub-contours. c In candidate contour C c Select the candidate contour C with the largest area from the sub-contours. d , will C b C c C d Add to candidate contour set C candidateIn the process, the image S is finally obtained. rough Candidate contour set C candidate ={C a C b C c C d};

[0044] Using a polygon fitting algorithm to select the candidate contour set C for the screen region candidate Each contour in the dataset is processed to fit the corresponding polygon, resulting in a set of fitted polygons P = {P a ,P b ,P c ,P d}, where P a Candidate contour C a The corresponding polygon, P b Candidate contour C b The corresponding polygon, P c Candidate contour C c The corresponding polygon, P d Candidate contour C d The corresponding polygon;

[0045] Calculate the candidate profile set C using a heuristic scoring function. candidate The fit between each candidate contour and its corresponding polygon is calculated, and the contours are sorted according to the fit.

[0046] It should be understood that for image S rough Preprocessing can reduce image noise and enhance the edge features of the device screen.

[0047] First, images with excessively high resolution (i.e., images with a resolution greater than 1920×1080) are compressed to N² times the original size (N² = 0.25). Then, the compressed image is binarized, Gaussian filtered, and subjected to opening and closing operations to obtain the preprocessed image S. preprocess .

[0048] It should be understood that image S rough If the resolution is no greater than 1920×1080, no compression is needed; binarization, Gaussian filtering, and opening / closing operations are performed directly to obtain the preprocessed image S. preprocess .

[0049] It should be understood that using a heuristic scoring function on the candidate contour set C candidate Each candidate contour in the screen region is scored. The scoring process calculates the fit between the fitted polygon and the corresponding contour. The higher the fit, the closer the contour is to a quadrilateral (a rectangular screen generally appears as a quadrilateral in the image after perspective transformation), and the greater the probability that it is the true contour of the screen.

[0050] In one embodiment, the first heuristic scoring function is:

[0051]

[0052] α+β+γ=1,α=0.6,β=0.3,γ=0.1

[0053] in, Let be the fit between the j-th fitted polygon and the j-th candidate contour. Let be the area overlap ratio between the j-th candidate contour and the j-th polygon. Let be the ratio of the perimeter of the j-th candidate contour to that of the j-th polygon. Let α be the offset distance ratio between the j-th candidate contour and the j-th polygon, β be the area overlap ratio weight, β be the perimeter ratio weight, and γ be the offset distance ratio weight.

[0054] In one embodiment, the formula for analyzing the area overlap ratio of the j-th candidate contour and the j-th polygon is:

[0055]

[0056] Among them, S Pj∩C Represents the j-th polygon P j and the corresponding j-th candidate contour C j The area of ​​the intersecting region, S Pj∪C Represents the j-th polygon P j and the corresponding j-th candidate contour C j The area of ​​the merged region.

[0057] Among them, the area overlap ratio The larger the size, the larger the overlapping area between the two, and the higher the fit; conversely, the smaller the overlapping area, the lower the fit.

[0058] The analytical formula for the perimeter ratio of the j-th candidate contour and the j-th polygon is:

[0059]

[0060] Among them, L Pj Represents the j-th polygon P j The perimeter, L Cj Represents the j-th candidate contour C j The perimeter.

[0061] Among them, in L Cj >L Pj In the case of L Cj As the denominator, the perimeter is... The larger the size, the smaller the difference in circumference between the two, and the better the fit; conversely, the larger the difference in circumference between the two, the lower the fit.

[0062] The analytical formula for the offset distance ratio between the j-th candidate contour and the j-th polygon is as follows:

[0063]

[0064] Among them, D j To fit polygon P j Each pixel (x) v y v ) and distance candidate contour C j The nearest point on the contour line (x near ,y near The deviation distance, g is the fitted polygon P j The number of pixels, λ is the offset distance weight, μ is the minimum offset distance, and D k To fit polygon P k Each pixel (x) j y j ) and distance candidate contour C k The nearest point on the contour line (x near ,y near The deviation distance of k is given by k = a, b, c, d.

[0065] Among them, D j The smaller, The larger the value, the smaller the offset distance and the higher the fit.

[0066] In one embodiment, step S4 includes:

[0067] Step 41: Based on the sorted candidate contours, select the one with the highest fit f. 1st The first threshold T of the set fit a Compare, if f 1st ≥T a If f 1st <T a If so, the device screen shape category is determined to be a non-right-angle screen.

[0068] The first threshold can be set according to the actual situation, such as: T a =0.95.

[0069] It should be understood that if f 1st ≥T a This indicates that the fitted polygon and candidate contour have a high degree of fit, which conforms to the shape characteristics of a right-angled screen, thus the screen type is determined to be a right-angled screen; if f 1st <Ta Even the highest fit is low, indicating that the fitted polygon is difficult to fit the candidate contour very closely. It is highly likely that the screen type is a rounded screen, a notch screen, or a non-right-angle screen such as a screen with some edge information missing due to reflection during shooting.

[0070] Step 42: Compare the fit scores in descending order of fit, and calculate the fit score f that ranks first. 1st The fit with the second-ranked f 2nd The fit difference f diff f diff =f 1st -f 2nd and the fit difference f diff The second threshold T of the set fit b Make a comparison; if f diff >T b No further comparison is needed; if f diff ≤T b Compare the areas of the corresponding candidate contours and update the candidate contour with the larger area to f. 1st Then the updated first place f 1st and third place f 3rd Fourth place 4th Continue comparing the fit difference until f diff >T b Or complete with the fourth f 4th Comparison of fit difference; fit f 1st The corresponding candidate contour is used as the real device contour C. true .

[0071] Wherein, the second threshold T b It can be set according to the actual situation, such as: T b =0.02.

[0072] It should be understood that the candidate contour with the highest fit is very likely to be the actual screen contour. However, considering that the contour of the rectangular image being displayed on the screen may be regarded as a candidate contour and obtain a high fit, thus ranking ahead of the actual screen contour, the contour of the rectangular image being displayed on the screen can be eliminated through step 42.

[0073] Step 43, if the device screen shape category is a right-angled screen, then the actual device outline C is... true The image is restored proportionally by 1 / N^2 to its original value, thus restoring it to the original image S. rough Size, find the actual device outline C true The four points closest to the four vertices of its smallest bounding rectangle are taken as the four vertices of the screen area {p tl ,pbl ,p br ,p tr}, that is, to locate the area of ​​the screen in the image.

[0074] Step 44: If the device screen's shape category is a non-right-angle screen, perform line detection on the actual device outline C. true The minimum bounding rectangle region is processed to obtain the set of lines L = {l1,...,l...} that contain the screen boundary lines. u}, and form m quadrilateral frames B = {B1,...,B1,...} from the line segments in the set of lines L. m}, calculate the relationship between each quadrilateral frame and the actual device outline C. true The highest-rated quadrilateral frame B has the best fit. 1st The four vertices are used as the four vertices of the screen area {p tl ,p bl ,p br ,p tr}, that is, to locate the region of the screen in the image, where u is the total number of lines in the set of lines.

[0075] It should be understood that if the device screen type is a rounded corner screen, a notch screen, or a non-right-angle screen where reflections during shooting cause some edge information to be missing, on the one hand, bad pixels on the outline may affect the judgment of the real vertex (e.g., missing screen edge information leads to screen outline distortion, and points in the distorted part may be selected as vertices), which will directly affect the accurate positioning of the screen area; on the other hand, when using the (402) method for rounded corner screens, the four vertices of the screen will be located at the corner of the rounded corner screen, and the obtained screen area will be smaller than the real screen area. Step 44 can accurately locate the area position of the non-right-angle screen in the image.

[0076] In one embodiment, step 44 includes:

[0077] The actual device outline C true Expanding the smallest bounding rectangle region outward by a factor of N³, we obtain an image S that includes the entire device screen. L Using the LSD line detection algorithm, image S is detected. L The line segments in the image are filtered out, and line segments with a length less than a preset length are removed to obtain a set L of line segments that include the screen boundary lines;

[0078] A line merging algorithm based on angle difference and endpoint distance is used to merge line segments in the line segment set L that lie on a straight line, resulting in a merged line segment set L. merge ;

[0079] Using the line segment set L mergeLine segments in the diagram form angular structures, and angular structures that meet preset conditions are added to the angular structure set Q. orner ={Q1,...,Q q In the context of}, q represents the number of angle structures in the set of angle structures. The preset condition is that the angle formed by the intersection of the extensions of two line segments is between 60° and 120°, and the minimum distance from the intersection point to the two line segments is less than a third threshold T. c The angular structure, in which, e∈1, 2, ..., q, η is the distance weight, l min To form the angular structure Q e The length of the shortest line segment among the two line segments. Q is a angular structure e One of the line segments, Q is a angular structure e The other line segment;

[0080] The set of corner structures C orner The corner structures in the image are combined to form m quadrilateral frames, resulting in a set of quadrilateral frames B = {B1,...,B2}. m For every two quadrilaterals in the set B, calculate the overlap ratio (IoU). w≠h,w,h∈1,2,…,m,S Bw∩Bh Represents quadrilateral frame B w and quadrilateral frame B h The area of ​​the intersecting region, S Bw∪Bh Represents quadrilateral frame B w and quadrilateral frame B h The area of ​​the merged region, if IoU ≥ T d Remove the smaller box from B, T d The fourth threshold;

[0081] For the filtered set of quadrilateral frames B, calculate the value of each quadrilateral frame B in set B. w Contour of the actual device C true Area overlap ratio Comparison with perimeter Using the second heuristic scoring function to compare the quadrilateral frame with the actual device contour C true The fit was sorted, and the quadrilateral B with the highest fit was selected. 1st Define the precise area on the screen, and set the quadrilateral B. 1st The four vertices in the middle are used as the four vertices of the screen area {p tl ,p bl ,p br ,p tr}, that is, to locate the area of ​​the screen in the image.

[0082] Where N3 = 0.05.

[0083] Among them, the angular structure set C orner The way to combine the corner structures into m quadrilateral frames is as follows:

[0084] (1) Initialize the set of quadrilateral boxes B = {};

[0085] (2) Traversing the set C of angle structure orner and initialize set B w ={}, line={}; will set the current Q e In Add to Line

[0086] (3) Find the angular structure that shares a line segment with line. New line segment Add to line, and combine identical line segments Remove B from line and add B. w middle

[0087] (4) Repeat (3) to obtain Find angular structures that share two identical line segments with line. Line segment Join B w In the middle, we obtained (l a w ≠l b w ≠l c w ≠l d w ), B w Add to B;

[0088] Repeat steps (2) to (4) until the traversal is complete, resulting in B = {B1...B}. m}

[0089] It should be understood that due to missing screen boundary information, notch screens, etc., the screen boundary will be divided into at least two line segments. When using a line detection algorithm, this boundary will be detected as multiple line segments on the same straight line, without a complete screen border line segment. To solve this problem, a line merging algorithm based on angle difference and endpoint distance is used to merge line segments on the same straight line, resulting in a merged line segment set L. merge .

[0090] It should be understood that if IoU≥T d Removing smaller boxes from B can eliminate duplicate quadrilateral boxes, reducing the computational load in subsequent processing.

[0091] In one embodiment, the second heuristic scoring function is:

[0092]

[0093] α + β = 1, α = 0.7, β = 0.3

[0094] in, Quadrilateral B w Contour of the actual device C true The fit.

[0095] Wherein, the w-th quadrilateral frame B w And the actual device outline C true The analytical formula for the area overlap ratio is:

[0096]

[0097] in, B represents the w-th quadrilateral frame. w And the actual device outline C true The area of ​​the intersecting region Represents the outline of the actual device C true And the actual device outline C true The area of ​​the merged region.

[0098] Among them, the area overlap ratio The larger the size, the larger the overlapping area between the two, and the higher the fit; conversely, the smaller the overlapping area, the lower the fit.

[0099] The w-th quadrilateral frame B w And the actual device outline C true The analytical formula for the perimeter ratio is:

[0100]

[0101] Among them, L Bw B represents the w-th quadrilateral frame. w The perimeter, L Ctrue Represents the outline of the actual device C true The perimeter.

[0102] Among them, in L Bw >L Ctrue In the case of L Bw As the denominator, the perimeter is... The larger the size, the smaller the difference in circumference between the two, and the better the fit; conversely, the larger the difference in circumference between the two, the lower the fit.

[0103] The aforementioned multi-type screen localization method in visual images can achieve precise localization of screen areas from images with high accuracy and robustness. It overcomes the problem that traditional localization methods based on color and edge information, as well as basic target detection algorithms, cannot accurately locate screens of various types of devices. Furthermore, by employing heuristic methods and line detection technology based on screen features, it can achieve precise localization of other screen types, such as right-angled screens, rounded-corner screens, notch screens, and screens lacking partial screen border information. It has a wide range of applications, not only for mobile phones and tablets, but also for laptops, desktop monitors, and even for devices such as automotive displays, ATMs, and flight instruments.

[0104] In one embodiment, a method for locating multiple types of screens in a visual image is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0105] Step 1: Construct a training dataset of 1500 images. Each sample in the dataset is an image of a mobile phone, tablet, laptop, desktop monitor, or in-vehicle display screen, containing the screen of a single device in an powered-on state. The label for each image is the device type and the mask of the area where the device is located. Next, train an instance segmentation model using the Mask R-CNN instance segmentation algorithm. Then, given an image S containing the complete device and its screen... input The trained instance segmentation model is used to identify image S. input The device in S, and find the device in S input The region M in which it is located.

[0106] Step 2: If the region M found in Step 1 is not empty, then find the smallest bounding rectangle R of that region. M To prevent R M The device screen is not fully covered, so R M Expanding outward by a factor of N1 (N1 = 0.15), we obtain the original image S. input Cropping out environmental interference around the device screen and including the entire device screen as much as possible in the image. rough ,like Figure 2 As shown; if the region M found in step 1 is empty, based on the characteristic that the device screen is generally located in the center of the captured image, according to the preset width τ×w and height τ×h S input The central region (w and h are respectively the image S) input The width and height (τ = 0.15) of the image S are cropped. input To minimize environmental interference around the device screen, an image S containing as much of the device screen as possible is obtained. rough ,like Figure 3 As shown.

[0107] Step 3: Process the extracted image S rough Multiple candidate screen contours are identified through edge detection. Based on the screen shape features, a heuristic method is used to fit polygons to all candidate contours. The candidate contours are scored and ranked according to the fit between the candidate contour and the fitted polygon.

[0108] In this embodiment, as Figure 4 As shown, the following preferred scheme can be used to obtain candidate contours in step 3 above:

[0109] (301) For the area M of the device screen obtained in step 2, since images with excessively high resolution will have a lot of noise, the image with excessively high resolution (the original image resolution is 4624×3472) is first compressed to 1 / 4 of its original resolution; for image S rough Preprocessing is performed to further reduce image noise and enhance the edge features of the device screen. Then, the compressed image is binarized, Gaussian filtered, and subjected to opening and closing operations to obtain the preprocessed image S. preprocess .

[0110] (302) Obtain the candidate contour set C for the corresponding device screen area. candidate First, the preprocessed image S... preprocess The Canny edge detection algorithm is used to detect all edges in the image, and the edges are connected by a closing operation to obtain image S. Canny Then, the contour extraction algorithm is used to extract image S. Canny The set of all contours in the set C = {C1,...,C2} n Next, we will calculate the C values ​​for each contour. i The area is calculated by counting the number of pixels contained in the contour, and the contour with the largest area is extracted as the first candidate contour C for the screen. a Add to candidate contour set C candidate In the , i = 1, ..., n.

[0111] (303)C a It could be the actual outline of the screen, or it could be the outline of the device bezel that encompasses the entire device area. Next, in C... a Three candidate contours (contours of three different colors: R, G, and B) are selected sequentially from the sub-contours of the current contour. The selection method is to choose the contour with the largest area from the sub-contours of the current contour as the next candidate contour. Specifically, in candidate contour C... a Select the candidate contour C with the largest area from the sub-contours. b ; in candidate contour C b Select the candidate contour C with the largest area from the sub-contours. c In candidate contour C cSelect the candidate contour C with the largest area from the sub-contours. d , will C b C c C d Add to candidate contour set C candidate In the end, C candidate ={C a C b C c C d}

[0112] In this embodiment, the following preferred scheme can be used to implement step 3 of scoring and sorting candidate contours:

[0113] (304) Use the polygon fitting algorithm to obtain the candidate contour set C of the screen region. candidate Each contour in the image is processed to fit its corresponding polygon, resulting in a set of fitted polygons P = {P a ,P b ,P c ,P d The fitting polygon algorithm works as follows: Each time, two points A and B in the contour are selected and connected by a straight line AB, which is used as the chord of a certain curve segment in the contour. The point C on this curve that is furthest from this straight line segment is obtained, and the distance d between it and AB is calculated. This distance d is compared with a pre-set threshold. If d is greater than the threshold, the curve is divided into two segments AC and BC using point C, and the above processing is performed on each segment. If the distance d is less than the threshold, the straight line segment AB is used as an approximation of this curve segment, and the processing of this curve segment on the contour is complete.

[0114] (305) Using a heuristic scoring function to evaluate candidate contours C candidate The candidate contours for each screen region are scored. The scoring process calculates the fit between the fitted polygon and the corresponding candidate contour. The higher the fit, the closer the candidate contour is to a quadrilateral (a rectangular screen generally appears as a quadrilateral in the image after perspective transformation), and the greater the probability that it is the true candidate contour of the screen.

[0115] Let the candidate contour be C j (C j ∈C candidate The fitted polygon is P. j (P j ∈P), fit is calculated from three perspectives:

[0116] (a) The area overlap ratio between the j-th candidate contour and the j-th polygon S Pj∩C Represents the j-th polygon P j and the corresponding j-th candidate contour Cj The area of ​​the intersecting region, S Pj∪C Represents the j-th polygon P j and the corresponding j-th candidate contour C j The area of ​​the merged region, The larger the value, the greater the overlapping area and the higher the fit. Figure 5 As shown in (b), candidate contour C d and fitted polygon P d Area overlap ratio Conversely, the smaller the overlapping area, the lower the fit, such as... Figure 5 As shown in (a), candidate contour C a and fitted polygon P a Area overlap ratio =0.9325.

[0117] (b) Perimeter ratio of candidate contour to fitted polygon L Pj Represents the j-th polygon P j The perimeter, L Cj Represents the j-th candidate contour C j circumference, The larger the value, the smaller the difference in circumference between the two, and the higher the fit. Figure 5 As shown in (b), candidate contour C d and fitted polygon P d The perimeter is greater than Conversely, the greater the difference in circumference between the two, the lower the fit. Figure 5 As shown in (a), candidate contour C a and fitted polygon P a The perimeter is greater than

[0118] (c) Deviation distance ratio in To fit polygon P j Each pixel (x) v y v ) and distance candidate contour C j The nearest point on the contour line (x near ,y near The deviation distance of D) j The smaller, The larger the value, the smaller the offset distance and the higher the fit. Figure 5 As shown in (b), candidate contour C d and fitted polygon P d Deviation distance ratio Conversely, the greater the deviation distance, the lower the fit, such as Figure 5 As shown in (a), candidate contour Ca and fitted polygon P a Deviation distance ratio

[0119] Using the first heuristic scoring function A comprehensive score is obtained to represent the fit between the fitted polygon and the candidate contour on the screen. For example... Figure 5 As shown, Figure 5 (a) Candidate contour C a and fitted polygon P a The fit is f 4th =0.8643, Figure 5 Candidate contour C in Figure (b) d and fitted polygon P d The fit is f 1st =0.9876, which clearly shows Figure 5 The contour in Figure (b) fits the fitted quadrilateral better, making it more likely to be the true contour of the screen. The first heuristic scoring function formula is:

[0120]

[0121] α+β+γ=1,α=0.6,β=0.3,γ=0.1

[0122] Step 4: Based on the sorted candidate contours obtained in Step 3, first determine the shape category of the device screen, including right-angle screens, rounded corner screens, notch screens, screens with missing edge information due to reflections during shooting, and other screen shapes. Based on this, different algorithms are used to locate the screen area and obtain the coordinates of the four vertices of the screen. The quadrilateral area formed by the four vertices is the finally identified and located device screen area.

[0123] In this embodiment, the following preferred solution can be used to implement step 4 above, which involves determining the screen type and locating it:

[0124] (401) Determine the device screen shape type. Sort the fit scores obtained in step (304) in descending order, and compare the maximum fit score f. 1st With the set threshold T a (T a =0.95) to determine the screen type. If f 1st ≥T a This indicates that the fitted polygon and the candidate contour have a high degree of fit, conforming to the characteristics of a right-angled screen shape (such as...). Figure 5 As shown in (b) above, candidate contour C d and fitted polygon P d Fit f 1st =0.9876), then the screen type is determined to be a right-angled screen; if f 1st<T a Even the highest fit is low, indicating that the fitted polygons are difficult to closely match the screen outline (e.g., Figure 6 As shown, candidate contour C c and fitted polygon P c Fit f 1st =0.9167), indicating that the screen type is highly likely to be a rounded corner screen, a notch screen, or a screen where some edge information is lost due to reflection during shooting, or other screen types.

[0125] (402) Locate the area of ​​the right-angled screen. For a right-angled screen, the contour with the highest fit is very likely to be the actual screen contour. However, considering that the contour of the rectangular image being displayed on the screen may be regarded as a candidate contour and obtain a high fit, thus ranking ahead of the actual screen contour, the following further processing is performed: The fit scores are compared in descending order, and the fit score f ranked first is calculated. 1st The fit with the second-ranked f 2nd The difference f diff =f 1st -f 2nd and with the set threshold T b (T b =0.02) for comparison; if f diff ≤T b This indicates that the difference in fit between the two candidate contours is negligible. In this case, compare the areas of their corresponding candidate contours and update the candidate contour with the larger area to f. 1st (This situation is as follows) Figure 7 As shown, Figure 7 (a) Candidate contour C is the precise location on the screen. b and fitted polygon P b Fit f 2nd =0.9950, Figure 7 (b) is a rectangular pattern inside the screen, candidate outline C c and fitted polygon P c Fit f 1st =0.9978); then update the first place f 1st and third place f 3rd Fourth place 4th Compare according to the above steps; if f diff >T b No further comparisons will be made; ultimately, the fit f will be determined. 1st The corresponding contour serves as the actual device contour C. true and C true Restore the image to its original size by a factor of 1 / N2.

[0126] The screen region will be located from the screen outline. The screen edge outline obtained by the outline extraction algorithm will fit the actual screen boundary more closely than the fitted polygon, so the four vertices of the screen will be marked on the screen outline. Find the actual screen outline C. true The four points closest to the four vertices of its smallest bounding rectangle are taken as the four vertices of the screen area {p tl ,p bl ,p br ,p tr}, that is, to locate the area of ​​the screen within the image, such as Figure 8 As shown.

[0127] (403) Locate the area of ​​the non-right-angle screen. If the device screen type is a rounded corner screen, a notch screen, or a screen where reflections during shooting cause some edge information to be missing, on the one hand, bad pixels on the outline may affect the judgment of the true vertex (e.g., missing screen edge information causes the screen outline to be distorted, and points in the distorted part may be selected as vertices, such as...). Figure 9 As shown), this will directly affect the accurate positioning of the screen area; on the other hand, when using the (402) method for rounded screens, the four vertices of the screen will be located at the corner of the rounded screen, and the obtained screen area will be smaller than the actual screen area, such as Figure 6 As shown. Therefore, for non-right-angled screens, this application does not select the vertices of the screen region from the screen outline, but instead uses line detection to obtain C from (402). true The minimum bounding rectangle region is processed to obtain the set of lines L = {l1,...,l...} that includes the screen boundary lines. u}, and form m quadrilateral frames B = {B1,...,B1} from the line segment set L. m}, In order to constitute B w The four line segments The intersection points on the extension lines are calculated according to (304) for each quadrilateral frame and C. true The highest-rated quadrilateral frame B has the best fit. 1st For a precise area of ​​the screen.

[0128] like Figure 10 As shown, the specific steps are as follows:

[0129] (a) Obtain the actual equipment outline C according to step (402). true , will C true Expand the minimum bounding rectangle region outward by a factor of N3 (N3 = 0.05) to obtain an image S that contains as much of the entire device screen as possible. L Using the LSD line detection algorithm, detect image S LThe line segments in the image are filtered out to remove those that are too short, resulting in a set L of line segments that include the screen boundary line. In cases where screen boundary information is missing or there is a notch, the screen boundary may be divided into multiple line segments. When using a line detection algorithm, this boundary will be detected as multiple line segments on the same straight line, without a complete screen border line segment. To solve this problem, a line merging algorithm based on angle difference and endpoint distance is used to merge line segments on the same straight line, resulting in a merged set L of line segments. merge ;

[0130] (b) Further expand the line segment set L merge The line segments in the text form a set of angular structures Q that satisfy the following preset conditions. orner ={Q1,...,Q q}, e∈1, 2, ..., q represents the angle formed by the intersection of the extensions of two line segments, which is between 60° and 120°, and the minimum distance from the intersection point to the two line segments is less than the third threshold. The corner structure. Next, the corner structure set C will be... orner The corner structures in the image are combined to form m quadrilateral frames, resulting in a set of quadrilateral frames B = {B1,...,B2}. m}, in, These are the four vertices of the quadrilateral frame. The next step is to examine every two quadrilateral frames in the set B. w≠h,w,h∈1,2,…,m,S Bw∩Bh Represents quadrilateral frame B w and quadrilateral frame B h The area of ​​the intersecting region, S Bw∪Bh Represents quadrilateral frame B w and quadrilateral frame B h The area of ​​the merged region, if IoU ≥ T d (T d =0.98), remove the smaller boxes from B (the larger boxes are more likely to contain screen areas). This process can remove duplicate boxes and reduce the amount of computation in subsequent processing.

[0131] (c) Next, for the filtered set of quadrilateral frames B, calculate the value of each quadrilateral frame B in B. j With C true Area overlap ratio Perimeter overlap ratio (To prevent excessive offset distance and D due to screen outline distortion, which could affect the accuracy of the fit, the offset distance ratio is not used here.) A second heuristic scoring function is used instead. α+β=1, α=0.7, β=0.3 for quadrilateral frame and outline C trueThe fit was scored and ranked, with the highest-scoring quadrilateral B being selected. 1st For a precise area of ​​the screen, such as Figure 11 As shown ( Figure 11 (a) B 1st The corresponding candidate box B1 and contour C true Fit f 1st =0.9435. Figure 11 (b) B 1st The corresponding candidate box B3 and contour C true Fit f 1st =0.9869). B 1st The four intersection points are used as the four vertices of the screen region {p tl ,p bl ,p br ,p tr}, that is, to locate the area of ​​the screen within the image, such as Figure 12 As shown.

[0132] The devices involved in this application can be used not only in mobile phones and tablets, but also in laptops, desktop monitors, vehicle displays, ATMs, flight instruments, and other devices.

[0133] In experiments using the aforementioned multi-screen localization method based on visual images, 1000 images were captured and tested from 12 devices, including a TIIDA automotive display, iPad 2019, MediaPad M5, Samsung Note 4, MECHREVO, and ThinkVision monitor. Actual testing showed that the screen of the tested device was accurately located in 90% of the tests, demonstrating that the aforementioned multi-screen localization method based on visual images can accurately locate the screen areas of various device types and has good performance.

[0134] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for locating multiple screen types in a visual image, used to identify device screens in an image and locate their respective areas, characterized in that, The method includes: Step S1: Use the instance segmentation algorithm of Mask R-CNN to identify image S input The device in the diagram is located in region M. Step S2: Process the region M so that the image of region M contains the entire device screen, and crop out the image S containing the entire device screen. rough ; Step S3: Process the image S rough Multiple candidate screen contours are identified by edge detection. Based on the screen shape features, a heuristic method is used to fit polygons to all candidate contours. The first heuristic scoring function is used to calculate the fit between each candidate contour and the corresponding polygon. The candidate contours are sorted in descending order of fit between the candidate contour and the fitted polygon. The first heuristic scoring function is: α+β+γ=1,α=0.6,β=0.3,γ=0.1 in, Let be the fit between the j-th fitted polygon and the j-th candidate contour. Let be the area overlap ratio between the j-th candidate contour and the j-th polygon. Let be the ratio of the perimeter of the j-th candidate contour to that of the j-th polygon. Let α be the offset distance ratio between the j-th candidate contour and the j-th polygon, β be the area overlap ratio weight, β be the perimeter ratio weight, and γ be the offset distance ratio weight. Step S4: Based on the sorted candidate contours, select the one with the highest fit f. 1st The first threshold T of the set fit a Compare, if f 1st ≥T a If f 1st <T a If the device screen is determined to be a non-right-angled screen, different positioning algorithms are used to locate the screen area, obtain the coordinates of the four vertices of the screen, and use the quadrilateral area formed by the four vertices as the final identification and positioning area of ​​the device screen.

2. The method according to claim 1, characterized in that, Step S2 includes: If the region M where the device is located, as determined in step S1, is not empty, determine the minimum bounding rectangle R of the region M. M ; The minimum bounding rectangle R of the region M. M Expanding outward by a factor of N1, we obtain the image S input By cropping out environmental interference around the device screen, an image S containing the entire device screen is obtained. rough ; If the area M where the device is located, as determined in step S1, is empty, based on the characteristic that the device screen is centered in the captured image, according to the preset image S with width τ×w and height τ×h... input The central region is cropped out of the image S. input Environmental interference around the outer edge of the device screen is analyzed to obtain an image S containing the entire device screen. rough Where w and h are the images S and S, respectively. input The width and height are given, and τ is the cutting ratio parameter, τ = 0.

15.

3. The method according to claim 2, characterized in that, The image S rough Multiple candidate screen contours were identified through edge detection. Based on the screen shape features, a heuristic method was used to fit polygons to all candidate contours, including: Analyze the image S rough If the resolution is greater than 1920×1080, then the image S... rough Compressed to the image S rough The image is obtained by multiplying the value by N². If the image size is not greater than 1920×1080, no further compression is needed. Then, the compressed image or the image S is further processed. rough Binarization, Gaussian filtering, and opening / closing operations are performed to obtain the preprocessed image S. preprocess Where N2 = 0.25; For the preprocessed image S preprocess The Canny edge detection algorithm is used to detect all edges in the image, and the edges are connected by a closing operation to obtain image S. Canny ; Image S was extracted using a contour extraction algorithm. Canny The set of all contours in the set C = {C1,...,C2} n }, where C1 is the first contour, C n Let n be the nth contour, where n is the total number of contours; Statistical analysis of each contour C i The contour C is calculated by the number of pixels contained within it. i Given the area of ​​the contour, extract the contour with the largest area from set C and use it as the first candidate contour C for the screen. a Add to candidate contour set C candidate In the context, i = 1, ..., n; In candidate contour C a Three candidate contours are selected sequentially from the sub-contours, where the selection method is as follows: In candidate contour C... a Select the candidate contour C with the largest area from the sub-contours. b In candidate contour C b Select the candidate contour C with the largest area from the sub-contours. c In candidate contour C c Select the candidate contour C with the largest area from the sub-contours. d , will C b C c C d Add to candidate contour set C candidate In the process, the image S is finally obtained. rough Candidate contour set C candidate ={C a C b C c C d }; Using a polygon fitting algorithm to select the candidate contour set C for the screen region candidate Each contour in the dataset is processed to fit the corresponding polygon, resulting in a set of fitted polygons P = {P a ,P b ,P c ,P d }, where P a Candidate contour C a The corresponding polygon, P b Candidate contour C b The corresponding polygon, P c Candidate contour C c The corresponding polygon, P d Candidate contour C d The corresponding polygon.

4. The method according to claim 3, characterized in that, The analytical formula for the area overlap ratio of the j-th candidate contour and the j-th polygon is as follows: in, Represents the j-th polygon P j and the corresponding j-th candidate contour C j The area of ​​the intersecting region Represents the j-th polygon P j and the corresponding j-th candidate contour C j The area of ​​the merged region; The analytical formula for the perimeter ratio of the j-th candidate contour and the j-th polygon is: Among them, L Pj Represents the j-th polygon P j The perimeter, L Cj Represents the j-th candidate contour C j circumference; The analytical formula for the offset distance ratio between the j-th candidate contour and the j-th polygon is as follows: Among them, D j To fit polygon P j Each pixel (x) v y v ) and distance candidate contour C j The nearest point on the contour line (x near ,y near The deviation distance, g is the fitted polygon P j The number of pixels, λ is the offset distance weight, μ is the minimum offset distance, and D k To fit polygon P k Each pixel (x) j y j ) and distance candidate contour C k The nearest point on the contour line (x near ,y near The deviation distance of k is given by k = a, b, c, d.

5. The method according to claim 4, characterized in that, The process involves using different positioning algorithms to locate the screen area, obtaining the coordinates of the four vertices of the screen, and using the quadrilateral region formed by the four vertices as the final identification and positioning area of ​​the device screen. Step 41: Compare the fit in descending order of fit, and calculate the fit f of the highest-ranked fit. 1st The fit with the second-ranked f 2nd The fit difference f diff f diff =f 1st -f 2nd and the fit difference f diff The second threshold T of the set fit b Make a comparison; if f diff >T b No further comparison is needed; if f diff ≤T b Compare the areas of the corresponding candidate contours and update the candidate contour with the larger area to f. 1st Then the updated first place f 1st and third place f 3rd Fourth place 4th Continue comparing the fit difference until f diff >T b Or complete with the fourth f 4th Comparison of fit difference; fit f 1st The corresponding candidate contour is used as the real device contour C. true ; Step 42, if the device screen shape category is a right-angled screen, then the actual device outline C is... true The image is restored proportionally by 1 / N^2 to its original value, thus restoring it to the original image S. rough Size, find the actual device outline C true The four points closest to the four vertices of its smallest bounding rectangle are taken as the four vertices of the screen area {p tl ,p bl ,p br ,p tr }, that is, to locate the area of ​​the screen in the image; Step 43: If the device screen's shape category is a non-right-angle screen, perform line detection on the actual device outline C. true The minimum bounding rectangle region is processed to obtain the set of lines L = {l1,...,l...} that contain the screen boundary lines. u }, and form m quadrilateral frames B = {B1,...,B1,...} from the line segments in the set of lines L. m }, calculate the relationship between each quadrilateral frame and the actual device outline C. true The highest-rated quadrilateral frame B has the best fit. 1st The four vertices are used as the four vertices of the screen area {p tl ,p bl ,p br ,p tr }, that is, to locate the region of the screen in the image, where u is the total number of lines in the set of lines.

6. The method according to claim 5, characterized in that, Step 44 includes: The actual device outline C true Expanding the smallest bounding rectangle region outward by a factor of N³, we obtain an image S that includes the entire device screen. L Using the LSD line detection algorithm, image S is detected. L The line segments in the image are filtered out, and line segments with a length less than a preset length are removed to obtain a set L of line segments that include the screen boundary lines; A line merging algorithm based on angle difference and endpoint distance is used to merge line segments in the line segment set L that lie on a straight line, resulting in a merged line segment set L. merge ; Using the line segment set L merge Line segments in the diagram form angular structures, and angular structures that meet preset conditions are added to the angular structure set Q. orner ={Q1,...,Q q In the context of}, q represents the number of angle structures in the set of angle structures. The preset condition is that the angle formed by the intersection of the extensions of two line segments is between 60° and 120°, and the minimum distance from the intersection point to the two line segments is less than a third threshold T. c The angular structure, in which, e∈1, 2, ..., q, η is the distance weight, l min To form the angular structure Q e The length of the shortest line segment among the two line segments. Q is a angular structure e One of the line segments, Q is a angular structure e The other line segment; The set of corner structures C orner The corner structures in the image are combined to form m quadrilateral frames, resulting in a set of quadrilateral frames B = {B1,...,B2}. m For every two quadrilaterals in the set B, calculate the overlap ratio (IoU). w≠h,w,h∈1,2,…,m, The area of ​​the region Represents quadrilateral frame B w and quadrilateral frame B h The area of ​​the merged region, if IoU ≥ T d Remove the smaller box from B, T d The fourth threshold; For the filtered set of quadrilateral frames B, calculate the value of each quadrilateral frame B in set B. w Contour of the actual device C true Area overlap ratio Comparison with perimeter Using the second heuristic scoring function to compare the quadrilateral frame with the actual device contour C true The fit was sorted, and the quadrilateral B with the highest fit was selected. 1st Define the precise area on the screen, and set the quadrilateral B. 1st The four vertices in the middle are used as the four vertices of the screen area {p tl ,p bl ,p br ,p tr }, that is, to locate the area of ​​the screen in the image.

7. The method according to claim 6, characterized in that, The second heuristic scoring function is: α+β＝1,α＝0.7,β＝0.3 in, Quadrilateral B w Contour of the actual device C true The fit.

Images