UI focus box identification method and device, and computer device

Abstract

This application provides a method, apparatus, and computer device for recognizing UI focus boxes. The recognition system acquires a focus box image and an image to be recognized, and decomposes the focus box image into four border sub-images. Then, using each border sub-image as a template, the OpenCV `matchTemplate` function is called to perform template matching on the image to be recognized, resulting in four template matching result matrices. The system uses the position of the corresponding border sub-image in each template matching result matrix within the focus box image as a reference, performs coordinate offsetting and data fusion on the four template matching result matrices, and obtains a final result matrix. The system iterates through the final result matrix and finds the first element with the maximum value. Finally, the matrix coordinates of the first element in the final result matrix are used as reference coordinates to calculate the overall coordinates of the focus box in the image to be recognized, thus achieving focus box recognition with fast detection speed and high accuracy.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus and computer device for recognizing a UI focus box. Background Technology

[0002] With the development of chip technology, TV main chips are becoming increasingly powerful, and correspondingly, TV software functions are becoming more complex. Traditional methods of relying on test engineers for manual testing of TVs are increasingly unable to meet product development schedule requirements, necessitating the introduction of automated testing. During automated testing, the automated testing device must simulate manual operation, issuing commands to the TV, and simultaneously check and determine the TV's status to decide the next action. In the TV's UI display, there are often multiple options. The currently valid option is highlighted in some way to distinguish it from other options. This is indicated by a colored border (e.g., white or red) around the icon thumbnail; this display method is known in the industry as the focus frame.

[0003] In automated testing, correctly detecting the position of the UI focus box and further identifying the content within it is like the "eyes" of the testing device, playing a crucial role in the correct execution of automated tests. Currently, there are relatively few implementations of automated UI focus box recognition in the industry. One approach uses corner detection combined with line detection, while another uses artificial neural networks. However, both of these methods require significant computation and have poor robustness. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method, apparatus, and computer device for recognizing a UI focus box.

[0005] To achieve the above objectives, this application provides a method for recognizing a UI focus box, including:

[0006] Acquire the focus frame image and the image to be identified, and decompose the focus frame image into 4 border sub-images;

[0007] Using each of the bounding sub-images as templates, the OpenCV matchTemplate function is called to perform template matching on the image to be recognized, resulting in template matching result matrices corresponding to each of the four bounding sub-images.

[0008] Using the position of the border sub-image corresponding to each of the template matching result matrices in the focus box image as a reference, coordinate offset and data fusion are performed on the four template matching result matrices to obtain the final result matrix;

[0009] Traverse all elements of the final result matrix and find the first element with the maximum value in the final result matrix;

[0010] Using the matrix coordinates of the first element in the final result matrix as reference coordinates, the overall coordinates of the focus box in the image to be identified are calculated, thereby realizing the identification of the focus box in the image to be identified.

[0011] This application also provides a UI focus box recognition device, including:

[0012] The disassembly module is used to acquire the focus frame image and the image to be identified, and to disassemble the focus frame image into four border sub-images;

[0013] The matching module is used to use each of the bounding sub-images as templates, call the OpenCV matchTemplate function to perform template matching on the image to be recognized, and obtain the template matching result matrix corresponding to each of the four bounding sub-images.

[0014] The processing module is used to take the position of the border sub-image corresponding to each of the template matching result matrices in the focus box image as a reference, perform coordinate offset and data fusion on the four template matching result matrices, and obtain the final result matrix.

[0015] The search module is used to traverse all elements of the final result matrix and find the first element that yields the maximum value in the final result matrix.

[0016] The calculation module is used to use the matrix coordinates of the first element in the final result matrix as reference coordinates to calculate the overall coordinates of the focus box in the image to be identified, thereby realizing the identification of the focus box in the image to be identified.

[0017] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0018] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0019] This application provides a method, apparatus, and computer device for recognizing a UI focus box. The recognition system acquires a focus box image and an image to be recognized, and decomposes the focus box image into four border sub-images. Then, using each border sub-image as a template, the OpenCV `matchTemplate` function is called to perform template matching on the image to be recognized, obtaining template matching result matrices corresponding to each of the four border sub-images. The recognition system uses the position of the corresponding border sub-image in the focus box image as a reference benchmark, performs coordinate offset and data fusion on the four template matching result matrices, and obtains the final result matrix. The recognition system iterates through all elements of the final result matrix and finds the first element with the maximum value. Finally, the matrix coordinates of the first element in the final result matrix are used as reference benchmark coordinates to calculate the overall coordinates of the focus box in the image to be recognized, thus realizing the recognition of the focus box in the image to be recognized. Compared with the prior art, the technical solution of this application does not require a large amount of data for model training, saving development investment and shortening development time. Furthermore, since no auxiliary changes are made to the image to be recognized during the processing, as much information from the original image is used as much as possible, information loss is avoided. The overall solution has a fast detection speed and high accuracy when implemented. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the steps of a UI focus box recognition method in one embodiment of this application;

[0021] Figure 2 This is an overall structural block diagram of the UI focus box recognition device in one embodiment of this application;

[0022] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0023] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] Reference Figure 1 One embodiment of this application provides a method for recognizing a UI focus box, including:

[0026] S1: Obtain the focus frame image and the image to be recognized, and decompose the focus frame image into 4 border sub-images;

[0027] S2: Using each of the bounding sub-images as templates, call the OpenCV matchTemplate function to perform template matching on the image to be recognized, and obtain the template matching result matrix corresponding to each of the four bounding sub-images;

[0028] S3: Using the position of the border sub-image corresponding to each of the template matching result matrices in the focus box image as a reference, coordinate offset and data fusion are performed on the four template matching result matrices to obtain the final result matrix;

[0029] S4: Traverse all elements of the final result matrix and find the first element with the maximum value in the final result matrix;

[0030] S5: Using the matrix coordinates of the first element in the final result matrix as reference coordinates, calculate the overall coordinates of the focus box in the image to be identified, thereby realizing the identification of the focus box in the image to be identified.

[0031] In this embodiment, the correction system retrieves a pre-recorded focus frame image, which serves as a template image for the focus frame. It also captures a screenshot from the driver (e.g., a TV) to obtain the image to be recognized. This image includes multiple icons (e.g., application icons), with the currently selected (or valid) icon highlighted around the focus frame. This focus frame has the same shape, size, and other specifications as the focus frame represented by the focus frame image. The focus frame is typically rectangular. The correction system divides the focus frame image into four border sub-images based on length and width. Using each border sub-image as a template and the image to be recognized as the original image to be matched, the correction system calls the `matchTemplate` function of the OpenCV library to perform template matching, thereby obtaining template matching result matrices corresponding to each of the four border sub-images. The value of each element in these four template matching result matrices represents the similarity between the corresponding border sub-image and the corresponding sub-image in the image to be recognized. The correction system uses the positions of the bounding box sub-images corresponding to each template matching structure matrix within the focus box image as reference benchmarks. It performs coordinate offsetting and data fusion on the four template matching result matrices to obtain the final result matrix. Specifically, the four template matching result matrices are the top bounding box template matching result matrix, the bottom bounding box template matching result matrix, the left bounding box template matching result matrix, and the right bounding box template matching result matrix. The top bounding box template matching result matrix corresponds to the template matching result of the top bounding box of the focus box on the image to be recognized; the bottom bounding box template matching result matrix corresponds to the template matching result of the bottom bounding box of the focus box on the image to be recognized; the left bounding box template matching result matrix corresponds to the template matching result of the left bounding box of the focus box on the image to be recognized; and the right bounding box template matching result matrix corresponds to the template matching result of the right bounding box of the focus box on the image to be recognized. The correction system shifts the bottom border template matching result matrix upwards by a first preset distance to obtain an offset bottom border template matching result matrix; and shifts the right border template matching result matrix to the left by a second preset distance to obtain an offset right border template matching result matrix. The first preset distance is the first vertical distance between the top and bottom borders of the focus frame, and the second preset distance is the second horizontal distance between the left and right borders of the focus frame. Then, using the top-left corner of the focus frame as a position reference, the correction system overlaps the top border template matching result matrix, the offset bottom border template matching result matrix, the left border template matching result matrix, and the offset right border template matching result matrix. Four overlapping matrices are selected at the overlapping positions, and a weighted sum is performed on these four overlapping matrices to obtain the final result matrix. The correction system iterates through all elements of the final result matrix and finds the first element with the maximum value. The matrix coordinates of this first element in the final result matrix are the coordinates of the top-left corner of the focus frame in the image to be recognized.The correction system uses this matrix coordinate system as a reference base coordinate system, and combines it with the first vertical distance and the second horizontal distance of the focus frame (i.e., the length and width of the focus frame) to calculate the coordinates of the lower left, upper right, and lower right corners of the focus frame. These coordinates together form the overall coordinates of the focus frame's location in the image to be recognized. The correction system then uses these overall coordinates to identify the position of the focus frame in the image.

[0032] Compared to existing technologies, the technical solution in this embodiment does not require a large amount of data for model training, saving development investment and shortening development time. Furthermore, since no auxiliary changes are made to the image to be recognized during the processing, as much information from the original image is used as possible, information loss is avoided. The overall solution achieves fast detection speed and high accuracy.

[0033] Furthermore, the focus frame is rectangular, and the four template matching result matrices are the top border template matching result matrix, the bottom border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix. The top border template matching result matrix corresponds to the template matching result of the top border of the focus frame on the image to be recognized; the bottom border template matching result matrix corresponds to the template matching result of the bottom border of the focus frame on the image to be recognized; the left border template matching result matrix corresponds to the template matching result of the left border of the focus frame on the image to be recognized; and the right border template matching result matrix corresponds to the template matching result of the right border of the focus frame on the image to be recognized. The step of using the position of the border sub-image corresponding to each of the template matching result matrices in the focus frame image as a reference, performing coordinate offset and data fusion on the four template matching result matrices to obtain the final result matrix includes:

[0034] S301: The lower border template matching result matrix is ​​shifted upward by a first preset distance to obtain an offset lower border template matching result matrix; and the right border template matching result matrix is ​​shifted to the left by a second preset distance to obtain an offset right border template matching result matrix, wherein the first preset distance is the first vertical distance between the upper and lower borders of the focus frame, and the second preset distance is the second horizontal distance between the left and right borders of the focus frame;

[0035] S302: Using the upper left corner of the focus frame as the position reference, overlap the upper border template matching result matrix, the lower border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix, and select the four overlapping matrices corresponding to the four overlapping positions of the matrices respectively;

[0036] S303: The four overlapping matrices are weighted and summed to obtain the final result matrix.

[0037] In this embodiment, the correction system selects the top-left corner of the focus frame as the offset reference to ensure the integrity of the offset matrix. Specifically, the correction system shifts the bottom border template matching result matrix upwards by a first preset distance to obtain an offset bottom border template matching result matrix; and shifts the right border template matching result matrix to the left by a second preset distance to obtain an offset right border template matching result matrix. The first preset distance is the first vertical distance between the top and bottom borders of the focus frame (i.e., the width of the focus frame), and the second preset distance is the second horizontal distance between the right and left borders of the focus frame (i.e., the length of the focus frame). After the coordinate offset is completed, the offset right border template matching result matrix coincides spatially with the left border template matching result matrix, and the offset bottom border template matching result matrix coincides spatially with the top border template matching result matrix. The correction system uses the top-left corner of the focus frame as the position reference. It overlaps the top border template matching result matrix, the offset bottom border template matching result matrix, the left border template matching result matrix, and the offset right border template matching result matrix, and takes the four overlapping matrices corresponding to the overlapping positions. The overlapping position of these four matrices represents the top-left corner region of the focus frame, and the size of this overlapping region is the width of the top border sub-image multiplied by the width of the left border sub-image. For example, the left border template matching result matrix is... The top border template matching result matrix is The offset right frame template matching result matrix is The offset bottom border template matching result matrix is The overlap matrix corresponding to the overlapping positions of the four matrices is 2*2, that is, the overlap matrix selected from the left border template matching result matrix is... The overlap matrix selected from the top border template matching result matrix is The overlap matrix selected from the offset right frame template matching result matrix is The overlap matrix selected from the offset bottom border template matching result matrix is The correction system performs a weighted summation on the four selected overlapping matrices according to preset weights, thereby obtaining the final result matrix after data fusion.

[0038] Furthermore, the step of performing a weighted summation of the four overlapping matrices to obtain the final result matrix includes:

[0039] S3031: Calculate the ratio between the first vertical distance and the second horizontal distance to obtain the first weight, and retrieve the preset second weight;

[0040] S3032: The overlapping matrices on the upper border template matching result matrix and the offset lower border template matching result matrix correspond to the second weight, and the overlapping matrices on the left border template matching result matrix and the offset right border template matching result matrix correspond to the first weight. The four overlapping matrices are weighted and summed to obtain the final result matrix.

[0041] In this embodiment, the overlapping matrices on the top border template matching result matrix and the offset bottom border template matching result matrix correspond to fixed weights, namely, a preset second weight (preferably 1). The weights corresponding to the overlapping matrices on the left border template matching result matrix and the offset right border template matching result matrix need to be calculated based on the actual length and width of the focus frame. Specifically, the correction system calculates the ratio between the first vertical distance and the second horizontal distance to obtain the required first weight. The correction system assigns the second weight to the overlapping matrices on the top border template matching result matrix and the offset bottom border template matching result matrix, and the first weight to the overlapping matrices on the left border template matching result matrix and the offset right border template matching result matrix. It then performs a weighted summation of the four overlapping matrices (i.e., multiplying the overlapping matrices on the top border template matching result matrix and the offset bottom border template matching result matrix by the second weight, and multiplying the overlapping matrices on the left border template matching result matrix and the offset right border template matching result matrix by the first weight) to obtain the final result matrix after data fusion.

[0042] Furthermore, the step of acquiring the focus frame image includes:

[0043] S101: Obtain the specification parameters input by the user;

[0044] S102: Obtain the corresponding focus frame image based on the specified parameters.

[0045] In this embodiment, the calibration system has a pre-built mapping table between specification parameters and focus frame images. This mapping table includes multiple sets of one-to-one corresponding specification parameters and focus frame images (different focus frame images have different shapes, lengths, and widths). The calibration system obtains the specification parameters manually input by the user (such as UI version information) and matches them with the focus frame image corresponding to the current specification parameters from the mapping table, so that the focus frame in the focus frame image is the same as the focus frame in the image to be recognized, thereby selecting the focus frame image as the matching template.

[0046] Furthermore, the matrix coordinates of the first element represent the coordinates of the upper left corner of the focal box. The step of using the matrix coordinates of the first element in the final result matrix as reference coordinates to calculate the overall coordinates of the focal box in the image to be recognized, thereby realizing the recognition of the focal box in the image to be recognized, includes:

[0047] S501: Using the matrix coordinates as the reference coordinates, and combining the first vertical distance downward translation, the coordinates of the lower left corner of the focus frame are obtained analytically; and combining the second horizontal distance to translate to the right, the coordinates of the upper right corner of the focus frame are obtained analytically.

[0048] S502: Using the coordinates of the lower left corner as a reference coordinate, and combining the second horizontal distance to translate to the right, the coordinates of the lower right corner of the focus frame are obtained by parsing; or, using the coordinates of the upper right corner as a reference coordinate, and combining the first vertical distance to translate downward, the coordinates of the lower right corner of the focus frame are obtained by parsing.

[0049] S503: By combining the coordinates of the upper left corner, the lower left corner, the upper right corner, and the lower right corner, the overall coordinates of the focus frame are obtained, and the position of the focus frame in the image to be identified is determined.

[0050] In this embodiment, the matrix coordinates of the first element represent the coordinates of the upper left corner of the focal box in the image to be recognized. The correction system uses these matrix coordinates as the reference coordinates and translates them downwards by a first vertical distance (i.e., translating downwards by a first vertical distance) to obtain the coordinates of the lower left corner of the focal box in the image to be recognized. Then, using a second horizontal distance, the matrix coordinates of the first element are translated to the right, and the translated coordinates are the coordinates of the upper right corner of the focal box in the image to be recognized. Next, the correction system uses the lower left corner coordinates as the reference coordinates and translates them to the right by a second horizontal distance to obtain the lower right corner of the focal box in the image to be recognized. Alternatively, the correction system uses the upper right corner coordinates as the reference coordinates and translates them downwards by a first vertical distance to obtain the coordinates of the lower right corner of the focal box. Thus, the correction system obtains the coordinates of the upper left, lower left, upper right, and lower right corners of the focal box in the image to be recognized. These coordinates, when combined, constitute the overall coordinates of the focal box, allowing the system to identify the position of the focal box in the image and complete the recognition of the focal box.

[0051] Furthermore, the focus frame image is rectangular, and the step of decomposing the focus frame image into four border sub-images includes:

[0052] S103: The focus frame image is divided into four borders according to the length and width of the outer contour, resulting in an upper border sub-image, a lower border sub-image, a left border sub-image, and a right border image. The lengths of the upper border sub-image and the lower border sub-image are the same as the length of the outer contour, and the lengths of the left border sub-image and the right border image are the same as the width of the outer contour.

[0053] In this embodiment, the focus frame image is rectangular. The correction system divides the focus frame image into four borders according to the length and width of the outer contour, resulting in an upper border sub-image, a lower border sub-image, a left border sub-image, and a right border sub-image. The lengths of the upper and lower border sub-images are the same as the length of the outer contour, while the lengths of the left and right border sub-images are the same as the width of the outer contour (i.e., if the upper, lower, left, and right border sub-images are reassembled into the focus frame image, overlapping areas will appear at the four corners of the focus frame image). This ensures the independence and integrity of each border sub-image, improving the accuracy and precision of subsequent template matching.

[0054] Reference Figure 2 This application provides a UI focus box recognition device, including:

[0055] The disassembly module 1 is used to acquire the focus frame image and the image to be identified, and to disassemble the focus frame image into 4 border sub-images;

[0056] Matching module 2 is used to call the OpenCV matchTemplate function to perform template matching on the image to be identified, using each of the bounding box sub-images as templates, to obtain the template matching result matrix corresponding to each of the four bounding box sub-images;

[0057] Processing module 3 is used to take the position of the border sub-image corresponding to each of the template matching result matrices in the focus frame image as a reference, perform coordinate offset and data fusion on the four template matching result matrices, and obtain the final result matrix;

[0058] Search module 4 is used to traverse all elements of the final result matrix and find the first element with the maximum value in the final result matrix;

[0059] The calculation module 5 is used to use the matrix coordinates of the first element in the final result matrix as reference coordinates to calculate the overall coordinates of the focus box in the image to be identified, thereby realizing the identification of the focus box in the image to be identified.

[0060] Furthermore, the focus box is rectangular, and the four template matching result matrices are respectively the top border template matching result matrix, the bottom border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix. The top border template matching result matrix corresponds to the template matching result of the top border of the focus box on the image to be recognized; the bottom border template matching result matrix corresponds to the template matching result of the bottom border of the focus box on the image to be recognized; the left border template matching result matrix corresponds to the template matching result of the left border of the focus box on the image to be recognized; and the right border template matching result matrix corresponds to the template matching result of the right border of the focus box on the image to be recognized. The processing module 3 includes:

[0061] The offset unit is used to shift the bottom border template matching result matrix upward by a first preset distance to obtain an offset bottom border template matching result matrix; and to shift the right border template matching result matrix to the left by a second preset distance to obtain an offset right border template matching result matrix, wherein the first preset distance is the first vertical distance between the top and bottom borders of the focus frame, and the second preset distance is the second horizontal distance between the left and right borders of the focus frame;

[0062] The fusion unit is used to make the upper border template matching result matrix, the offset lower border template matching result matrix, the left border template matching result matrix and the offset right border template matching result matrix coincide, taking the upper left corner of the focus box as the position reference, and selecting the four overlapping matrices corresponding to the four overlapping positions respectively;

[0063] The calculation unit is used to perform a weighted summation of the four overlapping matrices to obtain the final result matrix.

[0064] Furthermore, the computing unit includes:

[0065] The first calculation subunit is used to calculate the ratio between the first vertical distance and the second horizontal distance to obtain the first weight, and retrieve the preset second weight;

[0066] The second calculation subunit is used to perform a weighted summation of the four overlapping matrices, corresponding to the second weight on the overlapping matrices of the upper border template matching result matrix and the offset lower border template matching result matrix, and corresponding to the first weight on the overlapping matrices of the left border template matching result matrix and the offset right border template matching result matrix, to obtain the final result matrix.

[0067] Furthermore, the disassembly module 1 includes:

[0068] The acquisition unit is used to acquire the specification parameters input by the user;

[0069] A matching unit is used to match the corresponding focus frame image according to the specification parameters.

[0070] Furthermore, the matrix coordinates of the first element represent the coordinates of the upper left corner of the focus box. The calculation module 5 includes:

[0071] The first analysis unit is used to analyze the coordinates of the lower left corner of the focus frame by using the matrix coordinates as the reference coordinates and translating downwards by the first vertical distance; and to analyze the coordinates of the upper right corner of the focus frame by translating to the right by the second horizontal distance.

[0072] The second analysis unit is used to analyze the coordinates of the lower right corner of the focus frame by using the coordinates of the lower left corner as a reference coordinate and shifting to the right by the second horizontal distance; or, by using the coordinates of the upper right corner as a reference coordinate and shifting downward by the first vertical distance, to analyze the coordinates of the lower right corner of the focus frame.

[0073] The recognition unit is used to combine the coordinates of the upper left corner point, the lower left corner point, the upper right corner point, and the lower right corner point to obtain the overall coordinates of the focus box, and to identify the position of the focus box in the image to be recognized.

[0074] Furthermore, the focus frame image is rectangular, and the disassembly module 1 further includes:

[0075] The splitting unit is used to split the focus frame image into four borders according to the outer contour length and outer contour width, respectively, to obtain an upper border sub-image, a lower border sub-image, a left border sub-image, and a right border image. The lengths of the upper border sub-image and the lower border sub-image are the same as the outer contour length, and the lengths of the left border sub-image and the right border image are the same as the outer contour width.

[0076] In this embodiment, each module, unit, and subunit in the UI focus box recognition device is used to perform each step in the UI focus box recognition method described above, and the specific implementation process is not described in detail here.

[0077] This embodiment provides a UI focus box recognition device. The recognition system acquires a focus box image and an image to be recognized, and decomposes the focus box image into four border sub-images. Then, using each border sub-image as a template, the OpenCV `matchTemplate` function is called to perform template matching on the image to be recognized, obtaining template matching result matrices corresponding to each of the four border sub-images. The recognition system uses the position of the corresponding border sub-image in the focus box image as a reference benchmark, performs coordinate offset and data fusion on the four template matching result matrices, and obtains the final result matrix. The recognition system traverses all elements of the final result matrix and finds the first element with the maximum value. Finally, the matrix coordinates of the first element in the final result matrix are used as reference benchmark coordinates to calculate the overall coordinates of the focus box in the image to be recognized, thus realizing the recognition of the focus box in the image to be recognized. Compared with the prior art, the technical solution of this application does not require a large amount of data for model training, saving development investment and shortening development time. Furthermore, since no auxiliary changes are made to the image to be recognized in the processing flow, as much information in the original image as possible is used, thus avoiding information loss. The overall solution has a fast detection speed and high accuracy.

[0078] Reference Figure 3 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as focus box images. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for recognizing a UI focus box.

[0079] The processor executes the following steps of the UI focus box recognition method:

[0080] S1: Obtain the focus frame image and the image to be recognized, and decompose the focus frame image into 4 border sub-images;

[0081] S2: Using each of the bounding sub-images as templates, call the OpenCV matchTemplate function to perform template matching on the image to be recognized, and obtain the template matching result matrix corresponding to each of the four bounding sub-images;

[0082] S3: Using the position of the border sub-image corresponding to each of the template matching result matrices in the focus box image as a reference, coordinate offset and data fusion are performed on the four template matching result matrices to obtain the final result matrix;

[0083] S4: Traverse all elements of the final result matrix and find the first element with the maximum value in the final result matrix;

[0084] S5: Using the matrix coordinates of the first element in the final result matrix as reference coordinates, calculate the overall coordinates of the focus box in the image to be identified, thereby realizing the identification of the focus box in the image to be identified.

[0085] Furthermore, the focus frame is rectangular, and the four template matching result matrices are the top border template matching result matrix, the bottom border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix. The top border template matching result matrix corresponds to the template matching result of the top border of the focus frame on the image to be recognized; the bottom border template matching result matrix corresponds to the template matching result of the bottom border of the focus frame on the image to be recognized; the left border template matching result matrix corresponds to the template matching result of the left border of the focus frame on the image to be recognized; and the right border template matching result matrix corresponds to the template matching result of the right border of the focus frame on the image to be recognized. The step of using the position of the border sub-image corresponding to each of the template matching result matrices in the focus frame image as a reference, performing coordinate offset and data fusion on the four template matching result matrices to obtain the final result matrix includes:

[0086] S301: The lower border template matching result matrix is ​​shifted upward by a first preset distance to obtain an offset lower border template matching result matrix; and the right border template matching result matrix is ​​shifted to the left by a second preset distance to obtain an offset right border template matching result matrix, wherein the first preset distance is the first vertical distance between the upper and lower borders of the focus frame, and the second preset distance is the second horizontal distance between the left and right borders of the focus frame;

[0087] S302: Using the upper left corner of the focus frame as the position reference, overlap the upper border template matching result matrix, the lower border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix, and select the four overlapping matrices corresponding to the four overlapping positions of the matrices respectively;

[0088] S303: The four overlapping matrices are weighted and summed to obtain the final result matrix.

[0089] Furthermore, the step of performing a weighted summation of the four overlapping matrices to obtain the final result matrix includes:

[0090] S3031: Calculate the ratio between the first vertical distance and the second horizontal distance to obtain the first weight, and retrieve the preset second weight;

[0091] S3032: The overlapping matrices on the upper border template matching result matrix and the offset lower border template matching result matrix correspond to the second weight, and the overlapping matrices on the left border template matching result matrix and the offset right border template matching result matrix correspond to the first weight. The four overlapping matrices are weighted and summed to obtain the final result matrix.

[0092] Furthermore, the step of acquiring the focus frame image includes:

[0093] S101: Obtain the specification parameters input by the user;

[0094] S102: Obtain the corresponding focus frame image based on the specified parameters.

[0095] Furthermore, the matrix coordinates of the first element represent the coordinates of the upper left corner of the focal box. The step of using the matrix coordinates of the first element in the final result matrix as reference coordinates to calculate the overall coordinates of the focal box in the image to be recognized, thereby realizing the recognition of the focal box in the image to be recognized, includes:

[0096] S501: Using the matrix coordinates as the reference coordinates, and combining the first vertical distance downward translation, the coordinates of the lower left corner of the focus frame are obtained analytically; and combining the second horizontal distance to translate to the right, the coordinates of the upper right corner of the focus frame are obtained analytically.

[0097] S502: Using the coordinates of the lower left corner as a reference coordinate, and combining the second horizontal distance to translate to the right, the coordinates of the lower right corner of the focus frame are obtained by parsing; or, using the coordinates of the upper right corner as a reference coordinate, and combining the first vertical distance to translate downward, the coordinates of the lower right corner of the focus frame are obtained by parsing.

[0098] S503: By combining the coordinates of the upper left corner, the lower left corner, the upper right corner, and the lower right corner, the overall coordinates of the focus frame are obtained, and the position of the focus frame in the image to be identified is determined.

[0099] Furthermore, the focus frame image is rectangular, and the step of decomposing the focus frame image into four border sub-images includes:

[0100] S103: The focus frame image is divided into four borders according to the length and width of the outer contour, resulting in an upper border sub-image, a lower border sub-image, a left border sub-image, and a right border image. The lengths of the upper border sub-image and the lower border sub-image are the same as the length of the outer contour, and the lengths of the left border sub-image and the right border image are the same as the width of the outer contour.

[0101] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a method for recognizing a UI focus box. The method for recognizing a UI focus box specifically includes:

[0102] S1: Obtain the focus frame image and the image to be recognized, and decompose the focus frame image into 4 border sub-images;

[0103] S2: Using each of the bounding sub-images as templates, call the OpenCV matchTemplate function to perform template matching on the image to be recognized, and obtain the template matching result matrix corresponding to each of the four bounding sub-images;

[0104] S3: Using the position of the border sub-image corresponding to each of the template matching result matrices in the focus box image as a reference, coordinate offset and data fusion are performed on the four template matching result matrices to obtain the final result matrix;

[0105] S4: Traverse all elements of the final result matrix and find the first element with the maximum value in the final result matrix;

[0106] S5: Using the matrix coordinates of the first element in the final result matrix as reference coordinates, calculate the overall coordinates of the focus box in the image to be identified, thereby realizing the identification of the focus box in the image to be identified.

[0107] Furthermore, the focus frame is rectangular, and the four template matching result matrices are the top border template matching result matrix, the bottom border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix. The top border template matching result matrix corresponds to the template matching result of the top border of the focus frame on the image to be recognized; the bottom border template matching result matrix corresponds to the template matching result of the bottom border of the focus frame on the image to be recognized; the left border template matching result matrix corresponds to the template matching result of the left border of the focus frame on the image to be recognized; and the right border template matching result matrix corresponds to the template matching result of the right border of the focus frame on the image to be recognized. The step of using the position of the border sub-image corresponding to each of the template matching result matrices in the focus frame image as a reference, performing coordinate offset and data fusion on the four template matching result matrices to obtain the final result matrix includes:

[0108] S301: The lower border template matching result matrix is ​​shifted upward by a first preset distance to obtain an offset lower border template matching result matrix; and the right border template matching result matrix is ​​shifted to the left by a second preset distance to obtain an offset right border template matching result matrix, wherein the first preset distance is the first vertical distance between the upper and lower borders of the focus frame, and the second preset distance is the second horizontal distance between the left and right borders of the focus frame;

[0109] S302: Using the upper left corner of the focus frame as the position reference, overlap the upper border template matching result matrix, the lower border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix, and select the four overlapping matrices corresponding to the four overlapping positions of the matrices respectively;

[0110] S303: The four overlapping matrices are weighted and summed to obtain the final result matrix.

[0111] Furthermore, the step of performing a weighted summation of the four overlapping matrices to obtain the final result matrix includes:

[0112] S3031: Calculate the ratio between the first vertical distance and the second horizontal distance to obtain the first weight, and retrieve the preset second weight;

[0113] S3032: The overlapping matrices on the upper border template matching result matrix and the offset lower border template matching result matrix correspond to the second weight, and the overlapping matrices on the left border template matching result matrix and the offset right border template matching result matrix correspond to the first weight. The four overlapping matrices are weighted and summed to obtain the final result matrix.

[0114] Furthermore, the step of acquiring the focus frame image includes:

[0115] S101: Obtain the specification parameters input by the user;

[0116] S102: Obtain the corresponding focus frame image based on the specified parameters.

[0117] Furthermore, the matrix coordinates of the first element represent the coordinates of the upper left corner of the focal box. The step of using the matrix coordinates of the first element in the final result matrix as reference coordinates to calculate the overall coordinates of the focal box in the image to be recognized, thereby realizing the recognition of the focal box in the image to be recognized, includes:

[0118] S501: Using the matrix coordinates as the reference coordinates, and combining the first vertical distance downward translation, the coordinates of the lower left corner of the focus frame are obtained analytically; and combining the second horizontal distance to translate to the right, the coordinates of the upper right corner of the focus frame are obtained analytically.

[0119] S502: Using the coordinates of the lower left corner as a reference coordinate, and combining the second horizontal distance to translate to the right, the coordinates of the lower right corner of the focus frame are obtained by parsing; or, using the coordinates of the upper right corner as a reference coordinate, and combining the first vertical distance to translate downward, the coordinates of the lower right corner of the focus frame are obtained by parsing.

[0120] S503: By combining the coordinates of the upper left corner, the lower left corner, the upper right corner, and the lower right corner, the overall coordinates of the focus frame are obtained, and the position of the focus frame in the image to be identified is determined.

[0121] Furthermore, the focus frame image is rectangular, and the step of decomposing the focus frame image into four border sub-images includes:

[0122] S103: The focus frame image is divided into four borders according to the length and width of the outer contour, resulting in an upper border sub-image, a lower border sub-image, a left border sub-image, and a right border image. The lengths of the upper border sub-image and the lower border sub-image are the same as the length of the outer contour, and the lengths of the left border sub-image and the right border image are the same as the width of the outer contour.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0124] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, first object, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, first object, or method. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, first object, or method that includes that element.

[0125] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for recognizing a UI focus box, characterized in that, include: Acquire the focus frame image and the image to be identified, and decompose the focus frame image into 4 border sub-images; Using each of the bounding sub-images as templates, the OpenCV matchTemplate function is called to perform template matching on the image to be recognized, resulting in template matching result matrices corresponding to each of the four bounding sub-images. Using the position of the border sub-image corresponding to each of the template matching result matrices in the focus box image as a reference, coordinate offset and data fusion are performed on the four template matching result matrices to obtain the final result matrix; Traverse all elements of the final result matrix and find the first element with the maximum value in the final result matrix; Using the matrix coordinates of the first element in the final result matrix as reference coordinates, the overall coordinates of the focus box in the image to be identified are calculated, thereby realizing the identification of the focus box in the image to be identified; The focus box is a rectangle, and the four template matching result matrices are the top border template matching result matrix, the bottom border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix. The top border template matching result matrix corresponds to the result of template matching on the image to be recognized by the top border of the focus box. The lower border template matching result matrix corresponds to the template matching result of the lower border of the focus box on the image to be recognized; The left border template matching result matrix corresponds to the template matching result of the left border of the focus frame on the image to be recognized, and the right border template matching result matrix corresponds to the template matching result of the right border of the focus frame on the image to be recognized; the step of using the position of the border sub-image corresponding to each of the template matching result matrices in the image of the focus frame as a reference, and performing coordinate offset and data fusion on the four template matching result matrices to obtain the final result matrix includes: The bottom border template matching result matrix is ​​shifted upward by a first preset distance to obtain an offset bottom border template matching result matrix; and the right border template matching result matrix is ​​shifted to the left by a second preset distance to obtain an offset right border template matching result matrix, wherein the first preset distance is the first vertical distance between the top and bottom borders of the focus frame, and the second preset distance is the second horizontal distance between the left and right borders of the focus frame; Using the top left corner of the focus frame as the position reference, the top border template matching result matrix, the offset bottom border template matching result matrix, the left border template matching result matrix, and the offset right border template matching result matrix are overlapped, and the four overlapping matrices corresponding to the four overlapping positions are selected respectively. The four overlapping matrices are weighted and summed to obtain the final result matrix.

2. The method for recognizing the UI focus box according to claim 1, characterized in that, The step of weighted summing of the four overlapping matrices to obtain the final result matrix includes: Calculate the ratio between the first vertical distance and the second horizontal distance to obtain the first weight, and retrieve the preset second weight; The overlapping matrices on the top border template matching result matrix and the offset bottom border template matching result matrix correspond to the second weight, and the overlapping matrices on the left border template matching result matrix and the offset right border template matching result matrix correspond to the first weight. The four overlapping matrices are weighted and summed to obtain the final result matrix.

3. The method for recognizing the UI focus box according to claim 1, characterized in that, The step of acquiring the focus box image includes: Obtain the specifications input by the user; The corresponding focus frame image is obtained by matching the specified parameters.

4. The method for recognizing the UI focus box according to claim 1, characterized in that, The matrix coordinates of the first element represent the coordinates of the top-left corner of the focus box. The step of using the matrix coordinates of the first element in the final result matrix as a reference coordinate to calculate the overall coordinates of the focus box in the image to be recognized, thereby realizing the recognition of the focus box in the image to be recognized, includes: Using the matrix coordinates as a reference base coordinate, and combining the first vertical distance downward translation, the coordinates of the lower left corner of the focus frame are obtained analytically; and combining the second horizontal distance to the right translation, the coordinates of the upper right corner of the focus frame are obtained analytically. Using the coordinates of the lower left corner as a reference coordinate, and combining the second horizontal distance to shift to the right, the coordinates of the lower right corner of the focus frame are obtained; or, using the coordinates of the upper right corner as a reference coordinate, and combining the first vertical distance to shift downward, the coordinates of the lower right corner of the focus frame are obtained. By combining the coordinates of the top left corner, the bottom left corner, the top right corner, and the bottom right corner, the overall coordinates of the focus frame are obtained, and the position of the focus frame in the image to be identified is determined.

5. The method for recognizing the UI focus box according to claim 1, characterized in that, The focus frame image is rectangular, and the step of decomposing the focus frame image into four border sub-images includes: The focus frame image is divided into four borders according to the length and width of the outer contour, resulting in an upper border sub-image, a lower border sub-image, a left border sub-image, and a right border image. The lengths of the upper border sub-image and the lower border sub-image are the same as the length of the outer contour, and the lengths of the left border sub-image and the right border image are the same as the width of the outer contour.

6. A UI focus box recognition device, characterized in that, include: The disassembly module is used to acquire the focus frame image and the image to be identified, and to disassemble the focus frame image into four border sub-images; The matching module is used to use each of the bounding sub-images as templates, call the OpenCV matchTemplate function to perform template matching on the image to be recognized, and obtain the template matching result matrix corresponding to each of the four bounding sub-images. The processing module is used to take the position of the border sub-image corresponding to each of the template matching result matrices in the focus box image as a reference, perform coordinate offset and data fusion on the four template matching result matrices, and obtain the final result matrix. The search module is used to traverse all elements of the final result matrix and find the first element that yields the maximum value in the final result matrix. The calculation module is used to use the matrix coordinates of the first element in the final result matrix as reference coordinates to calculate the overall coordinates of the focus box in the image to be identified, thereby realizing the identification of the focus box in the image to be identified. The focus box is a rectangle, and the four template matching result matrices are the top border template matching result matrix, the bottom border template matching result matrix, the left border template matching result matrix, and the right border template matching result matrix. The top border template matching result matrix corresponds to the result of template matching on the image to be recognized by the top border of the focus box. The lower border template matching result matrix corresponds to the template matching result of the lower border of the focus box on the image to be recognized; The left border template matching result matrix corresponds to the template matching result of the left border of the focus box on the image to be recognized, and the right border template matching result matrix corresponds to the template matching result of the right border of the focus box on the image to be recognized. The processing module includes: The offset unit is used to shift the bottom border template matching result matrix upward by a first preset distance to obtain an offset bottom border template matching result matrix; and to shift the right border template matching result matrix to the left by a second preset distance to obtain an offset right border template matching result matrix, wherein the first preset distance is the first vertical distance between the top and bottom borders of the focus frame, and the second preset distance is the second horizontal distance between the left and right borders of the focus frame; The fusion unit is used to make the upper border template matching result matrix, the offset lower border template matching result matrix, the left border template matching result matrix and the offset right border template matching result matrix coincide, taking the upper left corner of the focus box as the position reference, and selecting the four overlapping matrices corresponding to the four overlapping positions respectively; The calculation unit is used to perform a weighted summation of the four overlapping matrices to obtain the final result matrix.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images