Image assertion method, electronic device and computer-readable storage medium

Abstract

The present application provides a picture assertion method, an electronic device and a computer-readable storage medium, and relates to the field of computer technology. The method includes: collecting a target picture and a template picture, performing a primary matching process on the target picture and the template picture, and determining an initial matching result; determining whether the initial matching result is within a preset similarity range, and if so, determining a target sub-picture from the target picture based on the initial matching result; performing a secondary matching process on the target sub-picture and the template picture to determine the target matching result, and determining a picture assertion result of the target picture based on the target matching result, and the picture assertion result is used to indicate whether the visual effect of the target picture on different terminal devices is consistent. The present application combines template matching and feature extraction, performs secondary matching through feature extraction processing, and performs secondary calibration on the template matching result obtained by the primary matching, thereby improving the accuracy of automated testing and reducing the false alarm rate of testing.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image assertion method, an electronic device, and a computer-readable storage medium. Background Art

[0002] As users' requirements for application experience continue to increase, ensuring the consistency and accuracy of the user interface and user experience becomes critical. Especially in the field of automated testing, image assertion strategy, as a key automated testing method, is widely used in user interface consistency checking to ensure that the user interface on different devices performs consistently and the visual output meets expectations.

[0003] The existing image assertion strategy mainly adopts pixel-based template matching technology (Open Source Computer Vision Library, OpenCV), which traverses and matches the template image on the target image, calculates the degree of match between the template at each possible position and the sub-image at that position, and compares the match value with the preset threshold to determine whether the match is passed. However, since the OpenCV template matching technology is based on pixel-by-pixel comparison, it is very sensitive to device changes. For example, changes in screen resolution, rendering differences, or image scaling will significantly affect the accuracy of the matching results, resulting in test failures due to pixel differences even if the user interface visually meets expectations. Therefore, in the process of cross-device automated testing, especially when performing visual consistency testing, the image assertion strategy based on OpenCV template matching technology has obvious limitations, which reduces the accuracy of automated testing and increases the false alarm rate of the test. Summary of the invention

[0004] The purpose of the present application is to provide an image assertion method, an electronic device and a computer-readable storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problem that the image assertion strategy in the prior art has obvious limitations, thereby reducing the accuracy of automated testing and increasing the false alarm rate of the test.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for asserting an image, the method comprising:

[0007] Collecting a target image and a template image, and performing a primary matching process on the target image and the template image to determine an initial matching result, wherein the initial matching result is used to indicate the similarity between the target image and the template image, wherein the target image and the template image are from different terminal devices, and the size of the target image is larger than the size of the template image;

[0008] Determine whether the initial matching result is within a preset similarity range, and if so, determine a target sub-image from the target image based on the initial matching result;

[0009] A secondary matching process is performed on the target sub-image and the template image to determine a target matching result, and based on the target matching result, a picture assertion result of the target image is determined, wherein the target matching result is used to indicate whether the target image and the template image are successfully matched, and the picture assertion result is used to indicate whether the visual effects of the target image on different terminal devices are consistent.

[0010] As a possible implementation manner, performing initial matching processing on the target image and the template image to determine an initial matching result includes:

[0011] Traversing each matching position, for the currently traversed matching position, determining the pixel value between the candidate area corresponding to the matching position and the template image, the candidate area being the image area of ​​the target image at the matching position covered by the template image;

[0012] Determining the similarity between the candidate area corresponding to each matching position and the template image based on the pixel values ​​between the candidate area corresponding to each matching position and the template image;

[0013] The similarity between the candidate area corresponding to each matching position and the template image is normalized to determine the initial matching result.

[0014] As a possible implementation manner, determining a target sub-picture from the target picture based on the initial matching result includes:

[0015] The picture corresponding to the target candidate area in the target picture is used as the target sub-picture, and the target candidate area is the candidate area with the highest similarity with the template picture among multiple candidate areas.

[0016] As a possible implementation manner, performing secondary matching processing on the target sub-image and the template image to determine the target matching result includes:

[0017] Preprocessing the target sub-image and the template image to obtain a processed sub-image and a processed template image;

[0018] Feature extraction is performed on the processed sub-image and the processed template image to determine the target matching result.

[0019] As a possible implementation manner, performing feature extraction processing on the processed sub-image and the processed template image to determine the target matching result includes:

[0020] Inputting the processed sub-image and the processed template image into a preset neural network model, and respectively extracting a first feature vector of the processed sub-image and a second feature vector of the processed template image through the preset neural network model;

[0021] A first cosine similarity between the first feature vector and the second feature vector is determined, and the target matching result is determined according to the first cosine similarity and a first matching threshold.

[0022] As a possible implementation manner, determining the target matching result according to the first cosine similarity and a first matching degree threshold includes:

[0023] If the first cosine similarity is greater than or equal to the first matching degree threshold, it is determined that the target image and the template image are matched successfully;

[0024] If the first cosine similarity is less than the first matching degree threshold, it is determined that the target image fails to match the template image.

[0025] As a possible implementation manner, performing feature extraction processing on the processed sub-image and the processed template image to determine the target matching result includes:

[0026] Inputting the processed sub-image and the processed template image into a preset multimodal contrastive learning model, and respectively extracting a third eigenvector of the processed sub-image and a fourth eigenvector of the processed template image through the preset multimodal contrastive learning model;

[0027] A second cosine similarity between the third feature vector and the fourth feature vector is determined, and the target matching result is determined according to the second cosine similarity and a second matching threshold.

[0028] As a possible implementation manner, after determining whether the initial matching result is within a preset similarity range, the method further includes:

[0029] If the initial matching result is not within the preset similarity range, determining that the initial matching result is greater than an upper limit value of the preset similarity range, or less than a lower limit value of the preset similarity range;

[0030] If the initial matching result is greater than the upper limit value, it is determined that the target image and the template image are matched successfully; otherwise, it is determined that the target image and the template image are matched unsuccessfully.

[0031] In a second aspect, an embodiment of the present application provides a picture assertion device, the device comprising:

[0032] A primary matching module, used for collecting a target image and a template image, and performing a primary matching process on the target image and the template image to determine an initial matching result, wherein the initial matching result is used to indicate the similarity between the target image and the template image, wherein the target image and the template image are from different terminal devices, and the size of the target image is larger than the size of the template image;

[0033] A determination module, configured to determine whether the initial matching result is within a preset similarity range, and if so, determine a target sub-image from the target image based on the initial matching result;

[0034] A secondary matching module is used to perform secondary matching processing on the target sub-image and the template image, determine the target matching result, and determine the image assertion result of the target image based on the target matching result. The target matching result is used to indicate whether the target image and the template image are successfully matched, and the image assertion result is used to indicate whether the visual effect of the target image on different terminal devices is consistent.

[0035] As a possible implementation, the initial matching module is specifically used to:

[0036] Traversing each matching position, for the currently traversed matching position, determining the pixel value between the candidate area corresponding to the matching position and the template image, the candidate area being the image area of ​​the target image at the matching position covered by the template image;

[0037] Determining the similarity between the candidate area corresponding to each matching position and the template image based on the pixel values ​​between the candidate area corresponding to each matching position and the template image;

[0038] The similarity between the candidate area corresponding to each matching position and the template image is normalized to determine the initial matching result.

[0039] As a possible implementation manner, the determining module is specifically used to:

[0040] The picture corresponding to the target candidate area in the target picture is used as the target sub-picture, and the target candidate area is the candidate area with the highest similarity with the template picture among multiple candidate areas.

[0041] As a possible implementation, the secondary matching module is specifically used for:

[0042] Preprocessing the target sub-image and the template image to obtain a processed sub-image and a processed template image;

[0043] Feature extraction is performed on the processed sub-image and the processed template image to determine the target matching result.

[0044] As a possible implementation, the secondary matching module is specifically used for:

[0045] Inputting the processed sub-image and the processed template image into a preset neural network model, and respectively extracting a first feature vector of the processed sub-image and a second feature vector of the processed template image through the preset neural network model;

[0046] A first cosine similarity between the first feature vector and the second feature vector is determined, and the target matching result is determined according to the first cosine similarity and a first matching threshold.

[0047] As a possible implementation, the secondary matching module is specifically used for:

[0048] If the first cosine similarity is greater than or equal to the first matching degree threshold, it is determined that the target image and the template image are matched successfully;

[0049] If the first cosine similarity is less than the first matching degree threshold, it is determined that the target image fails to match the template image.

[0050] As a possible implementation, the secondary matching module is specifically used for:

[0051] Inputting the processed sub-image and the processed template image into a preset multimodal contrastive learning model, and respectively extracting a third eigenvector of the processed sub-image and a fourth eigenvector of the processed template image through the preset multimodal contrastive learning model;

[0052] A second cosine similarity between the third feature vector and the fourth feature vector is determined, and the target matching result is determined according to the second cosine similarity and a second matching threshold.

[0053] As a possible implementation manner, the determining module is further used to:

[0054] If the initial matching result is not within the preset similarity range, determining that the initial matching result is greater than an upper limit value of the preset similarity range, or less than a lower limit value of the preset similarity range;

[0055] If the initial matching result is greater than the upper limit value, it is determined that the target image and the template image are matched successfully; otherwise, it is determined that the target image and the template image are matched unsuccessfully.

[0056] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image assertion method as described in any one of the first aspects above.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image assertion method as described in any one of the first aspects above are executed.

[0058] According to the image assertion method, electronic device and computer-readable storage medium of the embodiments of the present application, a target image and a template image are collected, and a primary matching process is performed on the target image and the template image to determine an initial matching result, and to determine whether the initial matching result is within a preset similarity range. If so, a target sub-image is determined from the target image based on the initial matching result; a secondary matching process is performed on the target sub-image and the template image to determine a target matching result, and based on the target matching result, an image assertion result of the target image is determined. According to an embodiment of the present application, a target image and a template image are obtained from different terminal devices, and a template matching method is used to perform an initial matching process on the target image and the template image to determine an initial matching result. The initial matching result is screened according to a preset similarity range. If the initial matching result is within the preset similarity range, a target sub-image is determined from the target image, and a deep learning model is used to perform feature extraction on the target sub-image and the template image to perform a secondary calibration on the initial matching result. Based on the target matching result obtained by the secondary matching, it is determined whether the visual effects of the target image on different terminal devices are consistent. This not only verifies the correctness and versatility of the automated test cases and improves the compatibility between devices, but also reduces the occurrence of mismatches, effectively improving the accuracy of automated testing and reducing the false alarm rate of testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of a process flow of a picture assertion method provided in an embodiment of the present application is shown;

[0061] Figure 2 A schematic diagram of a process for determining an initial matching result provided in an embodiment of the present application is shown;

[0062] Figure 3 A schematic diagram of a candidate area provided in an embodiment of the present application is shown;

[0063] Figure 4 A schematic diagram of a process for determining a target matching result provided in an embodiment of the present application is shown;

[0064] Figure 5 A schematic diagram of a process of a picture assertion method based on a neural network model provided in an embodiment of the present application is shown;

[0065] Figure 6 A schematic diagram of a process of a picture assertion method based on a multimodal contrastive learning model provided in an embodiment of the present application is shown;

[0066] Figure 7 A schematic diagram of the structure of a picture assertion device provided in an embodiment of the present application is shown;

[0067] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0069] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0070] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0071] During the device automation test process, the test will first be run on device A based on the automated test case. The automated test case can be recorded on device A, so a partial image can be captured from device A as a template image, and a full-screen image can be captured as a target image. The template image of device A and the target image of device A can be matched and passed. However, when the automated test case recorded on device A is used to run the test on device B, the human eye visual effects presented on the two terminal devices are consistent in the same scene of the same application, but due to the differences in devices, it is difficult to pass the test on device B, resulting in low device adaptability of the automated test case. The existing image assertion method compares the areas in the template and the original image one by one based on pixel points. This process is sensitive to device changes. For example, the screen resolution of the terminal device and the rendering differences of the graphics processing unit (GPU) may affect the final matching result. Therefore, in the device automation test scenario, the device adaptation effect is not very good, which leads to a decrease in the accuracy of the automated test and an increase in the false alarm rate of the test.

[0072] To address the above issues, the embodiments of the present application combine template matching and feature extraction, perform secondary matching through feature extraction processing, and perform secondary calibration on the template matching results obtained by the initial matching, which significantly improves the device adaptation effect, improves the accuracy of automated testing, and reduces the false alarm rate of testing.

[0073] Figure 1 A schematic diagram of a process flow of a picture assertion method provided in an embodiment of the present application is shown. Figure 1 As shown, the method specifically comprises the following steps:

[0074] S101, collecting target images and template images, and performing initial matching processing on the target images and the template images to determine initial matching results.

[0075] Optionally, in the device automation test, using target images and template images from different terminal devices and of different sizes can simulate the screen performance of multiple terminal devices in a real environment, and by matching images of different sizes and resolutions, the compatibility and consistency of the application on different terminal devices can be ensured. In the present application, the target image and the template image come from different terminal devices, and the size of the target image is larger than the size of the template image. Specifically, the target image and the template image are respectively screenshots from different terminal devices, for example, the template image is a screenshot from device A, and the target image is a screenshot from device B, and the size of the target image is larger than the size of the template image, for example, the target image can be a full-screen image, and the template image can be a partial image.

[0076] Optionally, record an automated test case Case on device A, and capture a partial image from the automated test case Case as a template image. In the same scenario of the same application, capture a full-screen image from device B as the target device to perform a run test on device B. Exemplarily, the target image and the template image can be initially matched based on a template matching algorithm to determine an initial matching result, and the initial matching result is used to indicate the similarity between the target image and the template image. The template matching algorithm is, for example, a template matching strategy based on openCV. The template matching strategy based on openCV is an effective method for finding a portion in an image that matches a given template. By comparing the template image and each portion in the target image, the most similar match is found, thereby determining the similarity between the target image and the template image, and obtaining the above-mentioned initial matching result.

[0077] S102: Determine whether the initial matching result is within a preset similarity range. If so, determine a target sub-image from the target image based on the initial matching result.

[0078] Optionally, the initial matching result obtained using the template matching algorithm is compared with a preset similarity range. If the initial matching result is within the preset similarity range, a target sub-image is cropped from the target image according to the initial matching result. The target sub-image includes an image area that is most similar to the template image, which can be determined by the best matching position in the initial matching result and the size of the template image. Exemplarily, the preset similarity range defines an interval between an upper and lower limit, which can be specifically determined based on an upper limit value A and a lower limit value B. The upper limit value A is, for example, 80%, and the lower limit value B is, for example, 50%. Accordingly, the preset similarity range is (50%, 80%). If the similarity between the target image indicated by the initial matching result and the template image is between 50% and 80%, for example, the initial matching result is 65%, that is, it is determined that the initial matching result is within the preset similarity range, then a target sub-image is determined from the target image according to the initial matching result, so as to perform secondary matching processing based on the target sub-image and the template image, and calibrate the initial matching result.

[0079] It should be noted that if the initial matching result is within the preset similarity range, it indicates that the match between the target image and the template image is not obvious. Therefore, on the basis of narrowing the search range through the initial matching, a secondary matching is performed based on the selected target sub-image and the template image to obtain more accurate matching information. Since the possible matching area is already known based on the initial matching result, there is no need to scan and match the entire target image during the secondary matching, which greatly reduces the amount of calculation. In addition, the secondary matching can be used as a verification mechanism to determine whether the initial matching result is accurate. Since the initial matching is greatly affected by the differences in receiving devices, selecting the target sub-image for secondary matching on the basis of the initial matching to improve the matching effect can ensure a more efficient, more accurate and more reliable image assertion result.

[0080] Optionally, if the initial matching result is not within the preset similarity range, the initial matching result is determined to be greater than the upper limit value of the preset similarity range, or less than the lower limit value of the preset similarity range; if the initial matching result is greater than the upper limit value, it is determined that the target image and the template image are matched successfully, otherwise, it is determined that the target image and the template image are matched unsuccessfully.

[0081] Exemplarily, if the initial matching result is not within the preset similarity range, it means that the initial matching result is either lower than the lower limit of the preset similarity range or higher than the upper limit of the preset similarity range. This indicates that there is high confidence or low confidence between the target image and the template image. In this case, even if a more complex secondary matching is performed, it will not provide additional value and it will be difficult to increase the chance of successful matching. Therefore, when the initial matching result is not within the preset similarity range, it can be directly determined whether the target image and the template image are successfully matched based on the initial matching result. There is no need for secondary matching, and it can also avoid wasting computing resources and time.

[0082] Continuing with the above-mentioned preset similarity range of (50%, 80%) as an example, if the initial matching result is not within the preset similarity range, for example, the initial matching result is 85%, then it is determined that the initial matching result is greater than the upper limit of the preset similarity range of 80%, and it can be directly determined that the target image and the template image are matched successfully. Conversely, if the initial matching result is 40%, then it is determined that the initial matching result is less than the lower limit of the preset similarity range of 50%, and it can be directly determined that the target image and the template image fail to match.

[0083] S103, performing secondary matching processing on the target sub-image and the template image to determine the target matching result, and determining the image assertion result of the target image according to the target matching result.

[0084] Optionally, the target matching result is used to indicate whether the target image and the template image are matched successfully, and the image assertion result is used to indicate whether the visual effects of the target image on different terminal devices are consistent. In the present application, on the basis of the initial matching based on the template matching strategy, the selected target sub-image and the template image can be subjected to secondary matching processing in combination with the deep learning feature extraction strategy, wherein the deep learning feature extraction strategy can be implemented based on a deep learning model such as a neural network model or a multimodal contrast learning model, and then determine whether the target image and the template image are matched successfully based on the target matching result obtained from the secondary matching process, thereby providing more accurate matching information and reducing possible misjudgments caused by the initial matching.

[0085] Optionally, through secondary matching processing, the similarity between the target sub-image and the template image is further verified to obtain the target matching result, and based on the target matching result, it is confirmed whether the visual effects of the target image on different terminal devices are consistent. If the match is successful, it means that the visual effects displayed on the two terminal devices are consistent, otherwise there may be differences. This significantly improves the accuracy and reliability of image assertions in device automation testing, which can not only effectively reduce the possible misjudgment during the initial matching, but also ensure the consistency of visual effects between different terminal devices.

[0086] Based on this, according to the image assertion method provided in the embodiment of the present application, the target image and the template image are obtained from different terminal devices, and the template matching method is used to perform initial matching processing on the target image and the template image to determine the initial matching result, and the initial matching result is screened according to a preset similarity range. If the initial matching result is within the preset similarity range, the target sub-image is determined from the target image, and the deep learning model is used to extract features of the target sub-image and the template image to perform secondary calibration on the initial matching result, and determine whether the visual effects of the target image on different terminal devices are consistent based on the target matching result obtained by the secondary matching. This not only verifies the correctness and universality of the automated test cases and improves the adaptability between devices, but also reduces the occurrence of mismatching, effectively improving the accuracy of automated testing and reducing the false alarm rate of testing.

[0087] Figure 2 The following is a flow chart of a method for determining an initial matching result provided by an embodiment of the present application. Figure 2 As shown, the above step S101 performs initial matching processing on the target image and the template image to determine the initial matching result, which specifically includes the following steps:

[0088] S201, traverse each matching position, and for the currently traversed matching position, determine the pixel value between the candidate area corresponding to the matching position and the template image.

[0089] Optionally, the candidate area is the image area where the target image is covered by the template image at the matching position. Figure 3 As shown in , in the target image, you can customize the partition size to determine the window size, or you can use the size of the template image as the window size and slide the window across the entire target image to traverse all possible matching positions. For each traversed matching position, determine a candidate area, which is the area of ​​the target image at the current matching position covered by the template image, such as Figure 3 As shown in the shaded area, it can be understood that the size of the candidate area is the same as the size of the template image, that is, when performing secondary matching, whether the neural network model or the multimodal contrast learning model is used for secondary matching, the size of the input template image and the target sub-image is the same.

[0090] Exemplarily, in this application, the target image and the template image are initially matched based on the template matching strategy of openCV. The purpose of the template matching strategy of openCV is to find the area similar to the template image in the target image, and then the image area with the highest similarity can be used as the target sub-image for secondary matching. Specifically, the template image and the target image are input into openCV, the template image and the target image are converted into grayscale images, and the grayscale template image is successively compared with the local target image corresponding to each candidate area. For each candidate area and template image, the pixel values ​​between each candidate area and the template image are extracted, and each pixel value is used to represent the degree of matching between the corresponding candidate area in the target image and the template image.

[0091] S202: Determine the similarity between the candidate area corresponding to each matching position and the template image based on the pixel values ​​between the candidate area corresponding to each matching position and the template image.

[0092] Exemplarily, the normalized square difference can be used to calculate the similarity between the template image and each candidate region. The normalized square difference is a method for measuring the similarity between two images or signals, and its value range is usually between [-1,1], where 1 indicates complete identity, 0 indicates no correlation, and -1 indicates complete negative correlation. Based on the definition of the normalized square difference, the normalized square difference R(x,y) is determined according to the pixel values ​​of the template image, the pixel values ​​of the target image where the candidate regions corresponding to each matching position are located, and the average grayscale value of the template image and the target image where the candidate regions corresponding to each matching position are located. The normalized square difference R(x,y) is used to reflect the degree of similarity between the template image and each candidate region. The closer the normalized square difference R(x,y) is to 1, the more similar the candidate regions corresponding to each matching position are to the template image.

[0093] Exemplarily, in addition to the above-mentioned normalized square difference strategy, image similarity calculation methods such as square difference matching method, correlation matching method, normalized correlation matching method, coefficient matching method, and normalized coefficient matching method can also be used. Among them, the square difference matching method directly calculates the square difference of the pixel value at each position between the template image and the target image, and the smaller the matching value, the higher the matching degree. The correlation matching method refers to using the multiplication operation between the template and the image to calculate the similarity, and the larger the matching value, the higher the matching degree. The normalized correlation matching method refers to standardizing the correlation matching method to eliminate the influence of linear brightness changes on the similarity calculation, and the matching value range is between [-1,1], and the larger the value, the higher the matching degree. The coefficient matching method refers to matching the relative value of the template to its mean with the correlation value of the image to its mean, and the matching value range is between [-1,1], 1 represents a perfect match, -1 represents a bad match, and 0 represents no correlation. The normalized coefficient matching method is a standardized operation of the coefficient matching method, and the matching value range is also between [-1,1], and the larger the value, the higher the matching degree.

[0094] S203: normalize the similarity between the candidate area corresponding to each matching position and the template image to determine an initial matching result.

[0095] Exemplarily, the normalization process is to normalize the similarities between the candidate areas corresponding to all matching positions and the template image within a specific range, such as [0, 1], to ensure the comparability of similarity metrics of different scales or types, and use the normalized similarity as the initial matching result.

[0096] Based on this, a template matching strategy based on openCV is used to perform initial matching on the target image and the template image, and the similarity between the template image and the current candidate area is effectively calculated. In particular, when the normalized square difference method is used, a more accurate and robust initial matching result can be obtained, so as to determine whether to perform secondary matching processing based on the initial matching result.

[0097] As a possible implementation manner, the above step S102 determines the target sub-picture from the target picture based on the initial matching result, including: taking the picture corresponding to the target candidate area in the target picture as the target sub-picture.

[0098] Optionally, the target candidate region is the candidate region with the highest similarity to the template image among the multiple candidate regions. Figure 3As shown, in the process of initially matching the target image and the template image and determining the initial matching result, multiple candidate regions are formed when the template image is used to traverse the target image, and the similarity between each candidate region and the template image is calculated respectively. In the case of multiple candidate regions, the present application finds the region in the target image that is most similar to the template image, that is, selects the candidate region with the highest similarity to the template image from the multiple candidate regions as the target sub-image, thereby reducing the possibility of mismatching. On this basis, the target sub-image and the template image are used for secondary matching processing, thereby improving the accuracy of the matching result.

[0099] Based on this, by selecting the candidate area with the highest similarity, the probability of false detection can be greatly reduced, that is, irrelevant objects can be avoided from being identified as target objects, and the area closest to the template image can be effectively found even in complex backgrounds or with partial occlusion. In addition, compared with performing fine matching directly on the entire target image, the method of performing rough matching first and then refining can significantly reduce the amount of calculation and improve processing speed.

[0100] Figure 4 A schematic diagram of a target matching result determination method provided in an embodiment of the present application is shown. Figure 4 As shown, the above step S103 performs secondary matching processing on the target sub-image and the template image to determine the target matching result, which specifically includes the following steps:

[0101] S401 , pre-processing the target sub-image and the template image to obtain a processed sub-image and a processed template image.

[0102] Exemplarily, the target sub-image and the template image are preprocessed to improve the image quality, reduce noise and redundant information, and make the subsequent feature extraction and matching process more accurate and efficient. Preprocessing includes but is not limited to: grayscale processing, histogram equalization processing, noise removal, image enhancement, edge detection, etc. Among them, grayscale processing is to convert the color image into a grayscale image to reduce the amount of calculation, because the grayscale image only contains brightness information, while the color image contains information of three channels of red, green and blue. Histogram equalization processing refers to adjusting the grayscale distribution of the image so that the contrast of the image is enhanced and it is easier to distinguish different objects and regions. Noise removal refers to using a filter (such as a Gaussian filter, a mean filter, etc.) to remove noise in the image to reduce interference with feature extraction. Edge detection refers to using an edge detection algorithm to highlight the contour information in the image to improve the detection accuracy of feature points. Finally, the processed sub-image and the processed template image obtained after preprocessing have improved image quality and are easier to perform subsequent feature extraction.

[0103] S402: Perform feature extraction on the processed sub-image and the processed template image to determine a target matching result.

[0104] Optionally, the main purpose of the feature extraction process is to extract distinguishing features from the preprocessed image, which can be used in the subsequent matching process to determine the target matching result. In the present application, deep learning strategies such as neural network models or multimodal contrast learning models can be used for feature extraction, by detecting feature points and generating feature vectors for each detected feature point. The feature vector is used to describe the image information around the feature point, and a matching algorithm is used to match the feature vectors in the processed sub-image with the feature vectors in the processed template image to find the most similar feature point pairs, and the final target matching result is determined based on reference indicators such as the number of matched feature point pairs, matching quality (such as matching distance, matching angle, etc.).

[0105] Optionally, the above step S402 specifically includes: inputting the processed sub-image and the processed template image into a preset neural network model, and extracting the first eigenvector of the processed sub-image and the second eigenvector of the processed template image through the preset neural network model; determining the first cosine similarity between the first eigenvector and the second eigenvector, and determining the target matching result based on the first cosine similarity and the first matching threshold.

[0106] Exemplarily, the preset neural network model is, for example, a MobileNetV3Small model. The Mobile NetV3Small model is a lightweight convolutional neural network model. After the processed sub-image and the processed template image are input into the MobileNetV3Small model, the MobileNetV3Small model will respectively extract the first eigenvector of the processed sub-image and the second eigenvector of the processed template image, and calculate the first cosine similarity between the first eigenvector and the second eigenvector. The first cosine similarity is an indicator for measuring the similarity in the directions of the first eigenvector and the second eigenvector, and its value range is [-1,1], where 1 indicates that the two vectors are exactly the same, 0 indicates that the two vectors are perpendicular (no correlation), and -1 indicates that the two vectors are completely opposite.

[0107] Optionally, the target matching result is determined according to the first cosine similarity and the first matching threshold, including: if the first cosine similarity is greater than or equal to the first matching threshold, it is determined that the target image and the template image are matched successfully; if the first cosine similarity is less than the first matching threshold, it is determined that the target image and the template image are matched unsuccessfully.

[0108] Exemplarily, the first matching threshold is 80%, that is, 0.8. Assuming that the first cosine similarity is 0.9, if it is determined that the first cosine similarity 0.9 is greater than the first matching threshold 0.8, it is considered that the target image and the template image are successfully matched, which means that the similarity between the target image and the template image is high enough.

[0109] Exemplarily, if the first cosine similarity is 0.6, it is determined that the first cosine similarity 0.6 is less than the first matching threshold 0.8, and it is considered that the target image and the template image fail to match, which means that the difference between the target image and the template image is large.

[0110] Figure 5 The flowchart of a method for image assertion based on a neural network model provided in an embodiment of the present application is shown. Figure 5 As shown, the target image is used as the main object of image recognition, and the template image is used as the reference object for matching with the target image. The target image and the template image are respectively input into openCV for initial matching. Specifically, the template matching function in the openCV library can be used to calculate the matching degree between the target image and the template image, and a specific similarity value is obtained to represent the matching degree between the target image and the template image. If the matching degree is greater than the lower limit value A of the preset similarity range, it is directly determined that the target image and the template image are matched. If the matching degree is less than the lower limit value B of the preset similarity range, it is directly determined that the target image and the template image fail to match. If the matching degree is greater than the lower limit value B and less than the upper limit value A, it is necessary to perform secondary matching processing on the target image and the template image. Specifically, a target sub-picture is extracted from the target picture, which is a local picture in the target picture that has the highest matching degree with the template picture. The target sub-picture and the template picture are input into a neural network model (MobileNetV3Small model) for feature extraction processing, and the similarity between the target sub-picture and the template picture is determined. The similarity is used as the matching degree and compared with a preset first matching degree threshold C, for example, 80%. If the matching degree is less than the first matching degree threshold C, it is determined that the target picture and the template picture fail to match. Conversely, if the matching degree is greater than or equal to the first matching degree threshold C, it is determined that the target picture and the template picture are matched successfully. Finally, based on the matching result, it is asserted whether the visual effects of the target picture on different terminal devices are consistent.

[0111] Based on this, by inputting the processed sub-image and the processed template image into a preset neural network model such as the MobileNetV3Small model, the high-level feature vectors of the processed sub-image and the processed template image can be effectively extracted, and the similarity between the processed sub-image and the processed template image can be evaluated by calculating the cosine similarity. This not only improves the matching accuracy, but also can cope with complex background changes and partial occlusion, and is particularly suitable for application scenarios that require high-precision image matching.

[0112] Optionally, the above step S402 specifically includes: inputting the processed sub-image and the processed template image into a preset multimodal contrast learning model, and extracting the third eigenvector of the processed sub-image and the fourth eigenvector of the processed template image respectively through the preset multimodal contrast learning model; determining the second cosine similarity between the third eigenvector and the fourth eigenvector, and determining the target matching result based on the second cosine similarity and the second matching threshold.

[0113] Exemplarily, the preset multimodal contrastive learning model is, for example, a contrastive learning model (Contrastive Language–Image Pre-training, CLIP). The multimodal contrastive learning model CLIP is a multimodal vision and text learning model that can learn from a large amount of image-text pair data so that the representation of the image and the text description associated therewith are similar in the feature space. In the present application, after the processed sub-image and the processed template image are input into the multimodal contrastive learning model CLIP, the multimodal contrastive learning model CLIP can extract a feature representation that can represent the image content from the image data, that is, the third eigenvector of the processed sub-image and the fourth eigenvector of the processed template image, and calculate the second cosine similarity between the third eigenvector and the fourth eigenvector to measure the similarity between the processed sub-image and the processed template image.

[0114] Optionally, the target matching result is determined according to the second cosine similarity and the second matching threshold, including: if the second cosine similarity is greater than or equal to the second matching threshold, it is determined that the target image and the template image are matched successfully; if the second cosine similarity is less than the second matching threshold, it is determined that the target image and the template image are matched unsuccessfully.

[0115] Exemplarily, the second matching threshold is 95%, that is, 0.95. Assuming that the second cosine similarity is 0.97, it is determined that the second cosine similarity 0.97 is greater than the second matching threshold 0.95, then it is considered that the target image and the template image are successfully matched, which means that the similarity between the two images, the target image and the template image, is high enough.

[0116] Exemplarily, if the second cosine similarity is 0.7, it is determined that the second cosine similarity 0.7 is less than the second matching threshold 0.95, and it is considered that the target image and the template image fail to match, which means that the difference between the target image and the template image is large.

[0117] Figure 6 The flowchart of a method for image assertion based on a multimodal contrastive learning model provided in an embodiment of the present application is shown. Figure 6 As shown, similarly, the target image and the template image are respectively input into openCV for initial matching. Specifically, the template matching function in the openCV library can be used to calculate the matching degree between the target image and the template image, and a specific similarity value is obtained to represent the matching degree between the target image and the template image. If the matching degree is greater than the lower limit value A of the preset similarity range, it is directly determined that the target image and the template image are matched. If the matching degree is less than the lower limit value B of the preset similarity range, it is directly determined that the target image and the template image fail to match. If the matching degree is greater than the lower limit value B and less than the upper limit value A, it is necessary to perform secondary matching processing on the target image and the template image. Specifically, a target sub-image is extracted from the target image, where the target sub-image is the local image in the target image that has the highest matching degree with the template image. The target sub-image and the template image are input into a multimodal contrastive learning model (CLIP model) for feature extraction processing, and the similarity between the target sub-image and the template image is determined. The similarity is used as the matching degree and compared with a preset second matching degree threshold D, for example, 95%. If the matching degree is less than the second matching degree threshold D, it is determined that the target image and the template image have failed to match. Otherwise, if the matching degree is greater than or equal to the second matching degree threshold D, it is determined that the target image and the template image have passed the matching. Finally, based on the matching result, it is asserted whether the visual effects of the target image on different terminal devices are consistent.

[0118] It should be noted that the similarity between the target image and the template image is calculated based on the solution combining OpenCV template matching and the multimodal contrast learning model. The overall process is similar to the solution based on the combination of OpenCV template matching and the neural network model. However, due to the difference in principles between the multimodal contrast learning model and the neural network model, the first matching threshold C corresponding to the neural network model and the second matching threshold D corresponding to the multimodal contrast learning model are also different, and the second matching threshold D is greater than the first matching threshold C.

[0119] Based on this, by using a preset multimodal contrastive learning model, such as the CLIP model, to extract the feature vectors of the image and calculate the cosine similarity between these feature vectors, it is determined whether the processed sub-image matches the processed template image. In this way, by taking advantage of multimodal contrastive learning, it is possible to establish a connection between data of different modalities, thereby performing a more accurate match.

[0120] As a possible implementation, combining Figure 5 and Figure 6 As shown, the embodiments of the present application provide two image assertion methods. The first solution is to combine template matching with a neural network model, and the second solution is to combine template matching with a multimodal comparative learning model. In this way, on the basis of template matching based on openCV, the matching results obtained by template matching are corrected in combination with deep learning feature extraction, which significantly improves the accuracy of equipment automation testing.

[0121] For example, the actual test matching results of the above two solutions and the original template matching solution in various scenarios are shown in Table 1 below:

[0122] Table 1 Test matching results

[0123] Scenario Template Image Target Image Template Matching Solution 1 Solution 2 1 A1 B 0.6909 0.9242 0.9862 2 A2 B 0.7211 0.8819 0.9747 3 C D1 0.7928 0.8862 0.9930 4 C D2 0.4946 0.8386 0.9419

[0124] Referring to Table 1 above, the MobileNetV3Small model in Solution 1 performs relatively well in high-precision image classification tasks and can be used in the assertion step of the poster scene in automated testing. The CLIP model in Solution 2 performs well in zero-sample learning tasks and can judge the similarity of two images from the perspective of semantic content without having seen images of a specific category, overcoming the low matching degree caused by simple screen resolution and GPU rendering differences. OpenCV template matching is more sensitive to scenes where the recommended poster is out of focus and enlarged, so combining OpenCV with the MobileNetV3Small model or OpenCV with the CLIP model significantly improves the accuracy of automated test cases in automated testing scenarios and improves the versatility of automated test cases on different devices.

[0125] Based on the same inventive concept, an image assertion device corresponding to the image assertion method is also provided in the embodiment of the present application. Since the principle of solving the problem by the image assertion device in the embodiment of the present application is similar to the above-mentioned image assertion method in the embodiment of the present application, the implementation of the image assertion device can refer to the implementation of the image assertion method, and the repeated parts will not be repeated.

[0126] Reference Figure 7, which is a schematic diagram of the structure of a picture assertion device provided in an embodiment of the present application, the picture assertion device 700 includes: a primary matching module 701, a determination module 702, and a secondary matching module 703, wherein:

[0127] The initial matching module 701 is used to collect the target image and the template image, and perform initial matching processing on the target image and the template image to determine an initial matching result, where the initial matching result is used to indicate the similarity between the target image and the template image. The target image and the template image are from different terminal devices, and the size of the target image is larger than the size of the template image.

[0128] A determination module 702 is used to determine whether the initial matching result is within a preset similarity range, and if so, determine a target sub-image from the target image based on the initial matching result;

[0129] The secondary matching module 703 is used to perform secondary matching processing on the target sub-image and the template image, determine the target matching result, and determine the image assertion result of the target image based on the target matching result. The target matching result is used to indicate whether the target image and the template image are successfully matched, and the image assertion result is used to indicate whether the visual effect of the target image on different terminal devices is consistent.

[0130] Based on this, according to the image assertion device of the embodiment of the present application, the target image and the template image are obtained from different terminal devices, and the template matching method is used to perform initial matching processing on the target image and the template image to determine the initial matching result, and the initial matching result is screened according to a preset similarity range. If the initial matching result is within the preset similarity range, the target sub-image is determined from the target image, and the deep learning model is used to extract features of the target sub-image and the template image to perform secondary calibration on the initial matching result, and whether the visual effects of the target image on different terminal devices are consistent is determined based on the target matching result obtained by the secondary matching. This not only verifies the correctness and universality of the automated test cases and improves the adaptability between devices, but also reduces the occurrence of mismatches, effectively improving the accuracy of automated testing and reducing the false alarm rate of testing.

[0131] In a possible implementation manner, the initial matching module 701 is specifically used to:

[0132] Traverse each matching position, and for the currently traversed matching position, determine the pixel value between the candidate area corresponding to the matching position and the template image, where the candidate area is the image area of ​​the target image at the matching position covered by the template image;

[0133] Determine the similarity between the candidate area corresponding to each matching position and the template image based on the pixel values ​​between the candidate area corresponding to each matching position and the template image;

[0134] The similarity between the candidate area corresponding to each matching position and the template image is normalized to determine the initial matching result.

[0135] In a possible implementation manner, the determination module 702 is specifically configured to:

[0136] The image corresponding to the target candidate region in the target image is used as the target sub-image, and the target candidate region is the candidate region with the highest similarity with the template image among the multiple candidate regions.

[0137] In a possible implementation manner, the secondary matching module 703 is specifically used to:

[0138] Preprocessing the target sub-image and the template image to obtain a processed sub-image and a processed template image;

[0139] Perform feature extraction on the processed sub-image and the processed template image to determine the target matching result.

[0140] In a possible implementation manner, the secondary matching module 703 is specifically used to:

[0141] Inputting the processed sub-image and the processed template image into a preset neural network model, and extracting a first eigenvector of the processed sub-image and a second eigenvector of the processed template image respectively through the preset neural network model;

[0142] A first cosine similarity between the first feature vector and the second feature vector is determined, and a target matching result is determined according to the first cosine similarity and a first matching threshold.

[0143] In a possible implementation manner, the secondary matching module 703 is specifically used to:

[0144] If the first cosine similarity is greater than or equal to the first matching threshold, it is determined that the target image and the template image are matched successfully;

[0145] If the first cosine similarity is less than the first matching threshold, it is determined that the target image and the template image fail to match.

[0146] In a possible implementation manner, the secondary matching module 703 is specifically used to:

[0147] Inputting the processed sub-image and the processed template image into a preset multimodal contrastive learning model, and respectively extracting the third eigenvector of the processed sub-image and the fourth eigenvector of the processed template image through the preset multimodal contrastive learning model;

[0148] A second cosine similarity between the third eigenvector and the fourth eigenvector is determined, and a target matching result is determined according to the second cosine similarity and a second matching threshold.

[0149] In a possible implementation manner, the determination module 702 is further configured to:

[0150] If the initial matching result is not within the preset similarity range, determining that the initial matching result is greater than an upper limit value of the preset similarity range, or less than a lower limit value of the preset similarity range;

[0151] If the initial matching result is greater than the upper limit value, it is determined that the target image and the template image are matched successfully, otherwise, it is determined that the target image and the template image are matched unsuccessfully.

[0152] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0153] The present application embodiment also provides an electronic device 800, such as Figure 8 As shown, it is a schematic diagram of the structure of an electronic device 800 provided in an embodiment of the present application, including: a processor 801, a memory 802, and optionally, a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device 800 is running, the processor 801 communicates with the memory 802 through the bus 803. When the machine-readable instructions are executed by the processor 801, the method steps in any of the above picture assertion methods are executed.

[0154] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method steps in any of the above picture assertion methods are executed.

[0155] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0156] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0157] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A picture assertion method, characterized in that: include: Collecting a target image and a template image, and performing a primary matching process on the target image and the template image to determine an initial matching result, wherein the initial matching result is used to indicate the similarity between the target image and the template image, wherein the target image and the template image are from different terminal devices, and the size of the target image is larger than the size of the template image; Determine whether the initial matching result is within a preset similarity range, and if so, determine a target sub-image from the target image based on the initial matching result; A secondary matching process is performed on the target sub-image and the template image to determine a target matching result, and based on the target matching result, a picture assertion result of the target image is determined, wherein the target matching result is used to indicate whether the target image and the template image are successfully matched, and the picture assertion result is used to indicate whether the visual effects of the target image on different terminal devices are consistent.

2. The method according to claim 1, characterized in that The performing initial matching processing on the target image and the template image to determine an initial matching result includes: Traversing each matching position, for the currently traversed matching position, determining the pixel value between the candidate area corresponding to the matching position and the template image, the candidate area being the image area of ​​the target image at the matching position covered by the template image; Determining the similarity between the candidate area corresponding to each matching position and the template image based on the pixel values ​​between the candidate area corresponding to each matching position and the template image; The similarity between the candidate area corresponding to each matching position and the template image is normalized to determine the initial matching result.

3. The method according to claim 2, characterized in that The determining a target sub-picture from the target picture based on the initial matching result includes: The picture corresponding to the target candidate area in the target picture is used as the target sub-picture, and the target candidate area is the candidate area with the highest similarity with the template picture among multiple candidate areas.

4. The method according to claim 1, characterized in that The performing secondary matching processing on the target sub-image and the template image to determine the target matching result includes: Preprocessing the target sub-image and the template image to obtain a processed sub-image and a processed template image; Feature extraction is performed on the processed sub-image and the processed template image to determine the target matching result.

5. The method according to claim 4, characterized in that The performing feature extraction processing on the processed sub-image and the processed template image to determine the target matching result includes: Inputting the processed sub-image and the processed template image into a preset neural network model, and respectively extracting a first feature vector of the processed sub-image and a second feature vector of the processed template image through the preset neural network model; A first cosine similarity between the first feature vector and the second feature vector is determined, and the target matching result is determined according to the first cosine similarity and a first matching threshold.

6. The method according to claim 5, characterized in that The determining the target matching result according to the first cosine similarity and a first matching degree threshold comprises: If the first cosine similarity is greater than or equal to the first matching degree threshold, it is determined that the target image and the template image are matched successfully; If the first cosine similarity is less than the first matching degree threshold, it is determined that the target image fails to match the template image.

7. The method according to claim 4, characterized in that The performing feature extraction processing on the processed sub-image and the processed template image to determine the target matching result includes: Inputting the processed sub-image and the processed template image into a preset multimodal contrastive learning model, and respectively extracting a third eigenvector of the processed sub-image and a fourth eigenvector of the processed template image through the preset multimodal contrastive learning model; A second cosine similarity between the third feature vector and the fourth feature vector is determined, and the target matching result is determined according to the second cosine similarity and a second matching threshold.

8. The method according to claim 1, characterized in that After determining whether the initial matching result is within a preset similarity range, the method further includes: If the initial matching result is not within the preset similarity range, determining that the initial matching result is greater than an upper limit value of the preset similarity range, or less than a lower limit value of the preset similarity range; If the initial matching result is greater than the upper limit value, it is determined that the target image and the template image are matched successfully; otherwise, it is determined that the target image and the template image are matched unsuccessfully.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the image assertion method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image assertion method according to any one of claims 1 to 8 are executed.

Images