Image recognition method, device, storage medium and electronic device

Abstract

The present specification discloses an image recognition method, device, storage medium and electronic device, which can collect screenshot images of the interface displayed when the target application is running, so that stable and distinguishable first feature points for characterizing the screenshot images can be extracted from the collected screenshot images, and stable and distinguishable second feature points for characterizing the standard graphics corresponding to the component elements to be identified can be extracted, and then the component elements to be identified can be identified from the screenshot images by matching the first feature points with the second feature points.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image recognition method, device, storage medium and electronic device. Background Art

[0002] At present, when performing automated testing on an application, due to the mechanism limitations of the built-in engine in the application, it is impossible to directly locate the position of each specific element (such as buttons, labels, etc.) contained in the interface currently displayed by the terminal device for testing. Therefore, when performing automated testing on this part of the application, it is only possible to capture the interface currently displayed on the terminal device screen through the interface provided by the operating system, and obtain a standard screenshot corresponding to the specified element to be tested, so as to determine the position information of the standard screenshot in the screenshot by comparing the standard screenshot with the screenshot obtained by capturing the interface, and generate a click operation instruction based on the determined position information to test the function corresponding to the specified element in the application.

[0003] However, due to the interference of factors such as game special effects (such as particle effects, light and shadow effects, physical effects, etc.) set in the interface displayed by some applications (such as various games), when testing these applications, the screenshots taken cannot be matched with the standard screenshots corresponding to the specified elements due to the interference of special effects, and thus the automated testing procedures for these applications cannot be carried out.

[0004] Therefore, how to accurately identify the standard screenshot corresponding to a specified element from the screenshot is a problem that needs to be solved urgently. Summary of the invention

[0005] This specification provides an image recognition method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides an image recognition method, including:

[0008] Capture the interface displayed when the target application is running and obtain a screenshot image;

[0009] Determine at least one component element that needs to be identified from the interface based on the screenshot image as a component element to be identified, and obtain a standard image corresponding to the component element to be identified;

[0010] According to a preset feature extraction dimension, each image feature point is determined from the screenshot image as each first feature point, and each image feature point is determined from the standard image as each second feature point, wherein the preset feature extraction dimension includes: a distribution of brightness values ​​corresponding to each pixel point contained in the image, intersections between each line contained in the image, and changes in pixel values ​​corresponding to each pixel point contained in the image;

[0011] Obtaining a matching result according to the similarity between each of the first feature points and each of the second feature points, wherein the matching result is used to indicate whether an image region corresponding to each of the first feature points in the screenshot image is an image region that matches the standard image;

[0012] The task is executed according to the matching result.

[0013] Optionally, according to a preset feature extraction dimension, determining each image feature point from the screenshot image as each first feature point, and determining each image feature point from the standard image as each second feature point specifically includes:

[0014] Superimposing the standard image and the screenshot image several times, and taking the image of the overlapping image area in the screenshot image of each superposition as each sub-image obtained by superposition, wherein the overlapping image area of ​​each superposition is different;

[0015] If it is determined that there is no sub-image in the sub-images whose similarity with the standard image exceeds a preset threshold, then, according to the preset feature extraction dimension, each image feature point is determined from the screenshot image as each first feature point, and each image feature point is determined from the standard image as each second feature point.

[0016] Optionally, the method further comprises:

[0017] If it is determined that there is a sub-image among the sub-images whose similarity with the standard image exceeds a preset threshold, the task is executed according to the position of the sub-image whose similarity with the standard image exceeds the preset threshold in the screenshot image.

[0018] Optionally, performing a task according to the matching result specifically includes:

[0019] If it is determined according to the matching result that the image area corresponding to each first feature point in the screenshot image is not an image area matching the standard image, then screening each first feature point according to the position of each first feature point in the screenshot image to obtain screened first feature points;

[0020] The task is executed according to the feature point matching results obtained based on the filtered first feature points.

[0021] Optionally, filtering the first feature points according to the position of each first feature point in the screenshot image to obtain the filtered first feature points specifically includes:

[0022] For each first feature point, determine other first feature points adjacent to the first feature point in the screenshot image as reference first feature points corresponding to the first feature point;

[0023] The first feature points are screened according to the distance between each first feature point and the reference first feature point corresponding to each first feature point.

[0024] Optionally, performing a task according to a feature point matching result obtained based on each first feature point after screening specifically includes:

[0025] Re-determining the screenshot image according to the image area corresponding to each of the filtered first feature points;

[0026] According to the preset feature extraction dimension, each image feature point is determined from the re-determined screenshot image as each third feature point;

[0027] The task is executed according to the matching results of the third feature points and the second feature points.

[0028] Optionally, performing a task according to the matching result specifically includes:

[0029] If it is determined according to the matching result that the image area corresponding to each of the first feature points in the screenshot image is an image area that matches the standard image, then determining the position of the center point of the image area corresponding to each of the first feature points in the screenshot image according to the position of each of the first feature points in the screenshot image;

[0030] According to the position of the center point, the component element to be identified is identified from the screenshot image, and a simulated touch operation is performed on the identified component element to be identified;

[0031] A test task for the target application is performed according to an interface displayed by the target application after responding to the simulated touch operation.

[0032] Optionally, performing a task according to the matching result specifically includes:

[0033] If it is determined according to the matching result that the image area corresponding to each of the first feature points in the screenshot image is not an image area matching the standard image, then obtaining an associated screenshot image corresponding to the screenshot image, wherein the associated screenshot image and the screenshot image are interfaces displayed at different times during the running of the target application;

[0034] According to a preset feature extraction dimension, determining each image feature point from the associated screenshot image as each associated feature point;

[0035] The task is executed according to the matching results of the associated feature points and the second feature points.

[0036] This specification provides an image recognition device, including:

[0037] The capture module is used to capture the interface displayed when the target application is running and obtain a screenshot image;

[0038] an acquisition module, used to determine at least one component element that needs to be identified from the interface based on the screenshot image as a component element to be identified, and acquire a standard image corresponding to the component element to be identified;

[0039] a determination module, configured to determine, according to a preset feature extraction dimension, each image feature point from the screenshot image as each first feature point, and to determine each image feature point from the standard image as each second feature point, wherein the preset feature extraction dimension includes: a distribution of brightness values ​​corresponding to each pixel point contained in the image, intersections between each line contained in the image, and changes in pixel values ​​corresponding to each pixel point contained in the image;

[0040] a matching module, configured to obtain a matching result according to the similarity between each of the first feature points and each of the second feature points, wherein the matching result is used to indicate whether the image area corresponding to each of the first feature points in the screenshot image is an image area matching the standard image;

[0041] An execution module is used to execute the task according to the matching result.

[0042] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0043] In the image recognition method provided in the present specification, the interface displayed when the target application is running is first captured to obtain a screenshot image, at least one component element that needs to be identified from the interface based on the screenshot image is determined as the component element to be identified, and the standard image corresponding to the component element to be identified is obtained, and according to the preset feature extraction dimension, each image feature point is determined from the screenshot image as each first feature point, and each image feature point is determined from the standard image as each second feature point. The preset feature extraction dimension includes: the distribution of brightness values ​​corresponding to each pixel point contained in the image, the intersection points between each line contained in the image, and the change of pixel values ​​corresponding to each pixel point contained in the image. According to the similarity between each first feature point and each second feature point, a matching result is obtained, and the matching result is used to indicate whether the image area corresponding to each first feature point in the screenshot image is an image area matching the standard image, and the task is executed according to the matching result.

[0044] It can be seen from the above method that a screenshot image of the interface displayed when the target application is running can be collected, so that stable and distinguishable first feature points for characterizing the screenshot image can be extracted from the collected screenshot image, and stable and distinguishable second feature points for characterizing the standard graphic can be extracted from the standard graphic corresponding to the component element to be identified, and then the component element to be identified can be identified from the screenshot image by matching the first feature points with the second feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0046] Figure 1 A schematic diagram showing the display effects of the component elements provided in this manual on a terminal device;

[0047] Figure 2 A flowchart of an image recognition method provided in this specification;

[0048] Figure 3 A schematic diagram of the template matching process provided in this specification;

[0049] Figure 4 A schematic diagram of the image recognition process provided in this specification;

[0050] Figure 5 A schematic diagram of an image recognition device provided in this specification;

[0051] Figure 6 A method corresponding to the Figure 1Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0053] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0054] At present, when performing automated testing on applications, due to the mechanism limitations of the built-in engine in the application (for example, in game programs, the game engine often uses rendering technologies different from traditional UI applications in order to improve rendering efficiency and visual effects, such as using hardware acceleration, custom shaders, 3D graphics rendering, etc., which makes it difficult for conventional UI automated testing tools, such as positioning methods based on the control tree structure, to directly identify and operate elements in the game), it is impossible to directly control each component element of the interface displayed on the terminal device when the application is running for testing, such as Figure 1 shown.

[0055] Figure 1 This is a schematic diagram of the display effect of the component elements provided in this manual on the terminal device.

[0056] exist Figure 1 In the displayed frame sequence, each frame is a screenshot of the interface displayed on the terminal device at different times when the application is running. The gray area on each frame is the halo effect rendered by the built-in engine of the application. Figure 1 It can be seen from the frame sequence that due to the special effects, the screenshots corresponding to the same component element at different times are obviously different. These differences make it impossible to accurately determine the position of the standard screenshot in the screenshot when performing automated testing on the application, resulting in the inability to simulate touch operations on the component elements in the application, and thus the inability to test the application.

[0057] Figure 2 The following is a flow chart of an image recognition method provided in this specification, comprising the following steps:

[0058] S201: capturing the interface displayed when the target application is running, and obtaining a screenshot image.

[0059] In this specification, the business platform can receive a test request sent by a user through a terminal device used by the user, determine the application to be tested as the target application, and then capture the interface displayed when the target application is running to obtain a screenshot image.

[0060] Of course, the business platform can also collect a screen recording video of the interface displayed by the target application within a specified time period as the target video, and use each frame of the target video as a screenshot image in turn. For example: if there are five frames of images in the target video, the business platform can first use the first frame of the image as a screenshot image, and identify the first frame of the target video. If the component to be identified cannot be accurately identified from the first frame of the target video, the business platform can continue to identify the second frame of the target video as a screenshot image, and so on.

[0061] In this specification, the execution entity used to implement the image recognition method may refer to a designated device set up in a business platform by a server, etc., or may refer to a terminal device such as a desktop computer, a laptop computer, etc. For the sake of ease of description, the image recognition method provided in this specification is described below using the server as an example of the execution entity.

[0062] S202: Determine at least one component element that needs to be identified from the interface based on the screenshot image as a component element to be identified, and obtain a standard image corresponding to the component element to be identified.

[0063] Furthermore, after obtaining the screenshot image, the server can determine, based on a test request for the target application sent by the user, at least one component element that needs to be identified from the interface displayed when the target application is running based on the screenshot image, as the component element to be identified, and obtain a standard image corresponding to the component element to be identified.

[0064] In addition, the server may also determine, for each component element included in the interface displayed when the target application is running, that the component element is a component element to be identified that needs to be identified from the interface based on the screenshot image.

[0065] In the above content, component elements may refer to visual elements and control parts that constitute the interface displayed by the target application to the user, such as: button components, text box components, label components, check box components, icon components, progress bar components, navigation bar components, toolbar components, etc.

[0066] In the above content, the standard image corresponding to the component element to be identified can be the image corresponding to the component element to be identified provided by the user before being rendered by the game engine, which can be understood as the original image corresponding to the component element to be identified without any special effects.

[0067] In the above content, there may be multiple methods for obtaining the standard image corresponding to the component element to be identified. For example, the standard image corresponding to the component element to be identified may be determined according to a test request for a target application sent by a user.

[0068] For another example, the standard image corresponding to the component element to be identified may be determined from the pre-saved standard images corresponding to the component elements.

[0069] S203: According to a preset feature extraction dimension, each image feature point is determined from the screenshot image as each first feature point, and each image feature point is determined from the standard image as each second feature point, wherein the preset feature extraction dimension includes: a distribution of brightness values ​​corresponding to each pixel point contained in the image, intersections between each line contained in the image, and changes in pixel values ​​corresponding to each pixel point contained in the image.

[0070] Furthermore, the server can determine each image feature point from the screenshot image as each first feature point according to a preset feature extraction dimension, and determine each image feature point from the standard image as each second feature point, and then identify the component elements to be identified from the screenshot image by matching each first feature point with each second feature point.

[0071] Among them, the above-mentioned preset feature extraction dimensions include: the distribution of brightness values ​​corresponding to each pixel point contained in the image, the intersection points between each line contained in the image, and the change of pixel values ​​corresponding to each pixel point contained in the image. The following is a detailed description of the method for the server to determine each graphic feature point from the screenshot image from each of the above-mentioned feature extraction dimensions.

[0072] Specifically, when the above-mentioned feature extraction dimension is the distribution of brightness values ​​corresponding to each pixel contained in the image, the server can convert the screenshot image to the hue saturation value (HSV) color space or the hue saturation lightness (HSL) color space to obtain the converted screenshot image, and then according to the distribution of the brightness values ​​of the converted screenshot image, the points corresponding to the local maximum value and / or the points corresponding to the minimum value of the brightness value distribution can be extracted from the converted screenshot image as the first feature points.

[0073] In addition, the server may also extract points corresponding to local maximum values ​​and / or points corresponding to minimum values ​​of the luminance value distribution from the converted screenshot image according to the distribution of the luminance values ​​of the converted screenshot image as first feature points.

[0074] When the feature extraction dimensions mentioned above are intersection points between lines contained in the image, the server may use intersection points where two or more lines contained in the screenshot image intersect as first feature points.

[0075] When the feature extraction dimension is the change in pixel value corresponding to each pixel point contained in the image, the server can determine the edge points as the first feature points according to the change in pixel value of each pixel point contained in the screenshot image.

[0076] In addition, the server determines each image feature point from the screenshot image according to a preset feature extraction dimension as a method for each first feature point. The server can use a Gaussian kernel with a standard deviation of different sizes to convolve the screenshot image to perform Gaussian blur processing on the screenshot image to obtain each processed screenshot image after Gaussian blur processing, wherein the blur degree of each processed screenshot image gradually increases.

[0077] Furthermore, the server can determine, for each processed screenshot image, image feature points of the processed screenshot image as first feature points based on a differential image between the processed screenshot image and other processed screenshot images that are adjacent to the processed screenshot image in the order of processing.

[0078] The method by which the server determines the difference image between the processed screenshot image and other processed screenshot images adjacent to the processed screenshot image in the processing sequence may be a Difference of Gaussian (DoG) algorithm.

[0079] In actual application scenarios, in order to improve the efficiency of identifying the component elements to be identified from the screenshot image, the server can also determine each image feature point from the screenshot image as each first feature point according to the preset feature extraction dimension, and determine each image feature point from the standard image as each second feature point, and then perform template matching on the standard image and the screenshot image to obtain the matching result between the standard image and the screenshot image, as shown in the following example. Figure 3 shown.

[0080] Figure 3 A schematic diagram of the template matching process provided in this specification.

[0081] Combination Figure 3 It can be seen that the server can overlay the standard image with the screenshot image several times (in Figure 3In the figure, the image in the dotted box is the standard image superimposed on the screenshot image, and the image in the solid line is the screenshot image), wherein, for each superposition, the server can use the image in the image area of ​​the screenshot image that overlaps with the standard image as each sub-image obtained by superposition.

[0082] Among them, the server can control the standard image to slide relative to the screenshot image (such as sliding one pixel horizontally or vertically) after superimposing the standard image with the screenshot image each time, and then superimpose the standard image with the screenshot image again, so that the overlapping image area is different each time.

[0083] Furthermore, after obtaining each sub-image, if the server determines that there is no sub-image in the sub-image whose similarity with the standard image exceeds a preset target similarity threshold, it can determine each image feature point from the screenshot image as each first feature point according to the preset feature extraction dimension, and determine each image feature point from the standard image as each second feature point.

[0084] In addition, if the server determines that there is a sub-image in each sub-image whose similarity with the standard image exceeds the preset target similarity threshold, then the position of the center point of the sub-image whose similarity with the standard image exceeds the preset target similarity threshold is determined based on the position of the sub-image whose similarity with the standard image exceeds the preset target similarity threshold in the screenshot image, and then the component element to be identified is identified from the screenshot image based on the position of the center point of the sub-image, and a simulated touch operation is performed on the identified component element to be identified, so that the test task for the target application can be performed based on the interface displayed by the target application after responding to the simulated touch operation.

[0085] S204: Obtain a matching result based on the similarity between the first feature points and the second feature points, wherein the matching result is used to indicate whether the image region corresponding to the first feature points in the screenshot image is an image region matching the standard image.

[0086] S205: Execute the task according to the matching result.

[0087] The server may obtain a matching result indicating whether the image area corresponding to each first feature point in the screenshot image is an image area matching the standard image according to the similarity between each first feature point and each second feature point. Figure 4 shown.

[0088] Figure 4 A schematic diagram of the image recognition process provided in this specification.

[0089] Combination Figure 4 It can be seen that the server can determine the target similarity threshold configured by the user based on the test request for the target application sent by the user. Further, the server can determine whether the similarity between each first feature point and each second feature point exceeds the above-mentioned target similarity threshold. If so, the server can obtain a matching result that the image area corresponding to each first feature point in the screenshot image is an image area that matches the standard image.

[0090] At this time, the server can determine the position of the center point of the image area corresponding to each first feature point in the screenshot image based on the position of each first feature point in the screenshot image, and identify the component elements to be identified from the screenshot image based on the position of the center point of the image area corresponding to each first feature point in the screenshot image, and perform simulated touch operations on the identified component elements to be identified, and then perform test tasks for the target application based on the interface displayed by the target application after responding to the simulated touch operation.

[0091] In addition, when the server determines that the similarity between each first feature point and each second feature point does not exceed the above-mentioned target similarity threshold, the server can filter each first feature point according to the position of each first feature point in the screenshot image to obtain the filtered first feature points, and then perform the task according to the feature point matching results obtained based on the filtered first feature points.

[0092] Specifically, the server can determine, for each first feature point, other first feature points adjacent to the position of the first feature point in the screenshot image as reference first feature points corresponding to the first feature point, and filter each first feature point according to the distance between each first feature point and the reference first feature point corresponding to each first feature point.

[0093] There are multiple methods for the server to screen each first feature point according to the distance between each first feature point and the reference first feature point corresponding to each first feature point. For example, the server can determine whether to screen out each first feature point according to whether the distance between the first feature point and the reference first feature point corresponding to the first feature point exceeds a preset distance threshold.

[0094] For another example: the server can also determine from each first feature point, several first feature points with the longest distance between the corresponding reference first feature points and screen out them. The number of first feature points that the server needs to screen out here can be set according to actual needs. Of course, it can also be determined by the server based on a preset ratio value and the total number of first feature points (for example: if the total number of first feature points is one hundred and the preset ratio value is twenty percent, the server can determine that the number of first feature points that need to be screened out is twenty).

[0095] Furthermore, after obtaining the filtered first feature points, the server can obtain the matching results of the filtered first feature points and the second feature points according to the similarities between the filtered first feature points and the second feature points (the matching results of the filtered first feature points and the second feature points here may refer to whether the image area corresponding to the filtered first feature points in the screenshot image is the matching result of the image area matching the standard image).

[0096] However, after screening each first feature point, the image area corresponding to each first feature point after screening will change in the screenshot image compared to the image area corresponding to each first feature point before screening, and since the image content contained in image areas of different sizes is different, the degree of focus on the local image area when determining each feature point according to different image areas is also different, and the accuracy of each feature point determined according to different image areas in representing the image content in the local image area is also different.

[0097] For example: when the feature points are determined based on the screenshot image for the first time, since the feature points are determined based on the entire image content of the screenshot image, the determined feature points are distributed in the entire screenshot image and are used to characterize all content information contained in the entire screenshot image. After the feature points determined for the first time are screened out, the remaining feature points correspond to partial image areas in the screenshot image. At this time, the amount of content information contained in the partial image areas in the screenshot image is less than the amount of content information contained in the entire screenshot image. Therefore, the feature points redetermined based on the partial image areas in the screenshot image are only distributed in the partial image areas in the screenshot image and are used to characterize the content information contained in the partial image areas in the screenshot image. As a result, the feature points redetermined based on the partial image areas in the screenshot image can more accurately characterize the content information contained in the partial image areas in the screenshot image.

[0098] Based on this, the server can also re-determine the screenshot image through the above-mentioned feature point acquisition method according to the image area corresponding to each filtered first feature point (that is, the image area corresponding to each filtered first feature point in the screenshot image is used as a new screenshot image), so that each image feature point can be determined from the re-determined screenshot image according to the preset feature extraction dimension as each third feature point, and then the task can be executed according to the matching results of each third feature point and each second feature point (the matching results of each third feature point and each second feature point here may refer to the matching results of whether the image area corresponding to each third feature point in the screenshot image is the image area that matches the standard image).

[0099] It can be seen from the above content that when the image feature points determined from the screenshot image cannot match the image feature points determined from the standard image, the server can also screen and refine the image feature points determined from the screenshot image to obtain a matching result, and the above screening and refinement process can be once or multiple times, that is, when the image area corresponding to each third feature point obtained after screening and refining the image feature points determined from the screenshot image is still not the image area matching the standard image in the screenshot image, the server can continue to screen each third feature point until the preset termination condition is met.

[0100] The above termination condition can be set according to actual needs. For example, when it is determined that the image area corresponding to each image feature point after screening in the screenshot image is an image area matching the standard image, the above termination condition can be regarded as being satisfied. For another example, when the number of times each image feature point is screened from the screenshot image reaches a preset threshold, the above termination condition can be regarded as being satisfied.

[0101] Furthermore, since the special effects displayed at different times by the component elements to be identified are different after being rendered by the built-in engine of the target application, the degree of interference of the special effects received by identifying the component elements to be identified from the screenshots corresponding to the interface displayed by the target application at different times is also different.

[0102] Therefore, when the above method still cannot identify the component element to be identified from the screenshot image, the server can also obtain the associated screenshot image corresponding to the screenshot image, and determine each image feature point from the associated screenshot image according to the preset feature extraction dimension as each associated feature point, and perform the task based on the matching result of each associated feature point and each second feature point (the matching result of each associated feature point and each second feature point here may refer to whether the image area corresponding to each associated feature point in the associated screenshot image is the matching result of the image area matching the standard image), wherein the above-mentioned associated screenshot image and the screenshot image are interfaces displayed at different times during the operation of the target application.

[0103] It can be seen from the above content that the server can collect screenshots of the interface displayed when the target application is running, so that stable and distinguishable first feature points for characterizing the screenshots can be extracted from the collected screenshots, and stable and distinguishable second feature points for characterizing the standard graphics corresponding to the component elements to be identified can be extracted, and then the component elements to be identified can be identified from the screenshots by matching the first feature points with the second feature points.

[0104] The above are one or more image recognition methods implemented in this specification. Based on the same idea, this specification also provides a corresponding image recognition device, such as Figure 5 shown.

[0105] Figure 5 A schematic diagram of an image recognition device provided in this specification includes:

[0106] The capture module 501 is used to capture the interface displayed when the target application is running and obtain a screenshot image;

[0107] An acquisition module 502 is used to determine at least one component element that needs to be identified from the interface based on the screenshot image as a component element to be identified, and to acquire a standard image corresponding to the component element to be identified;

[0108] A determination module 503 is used to determine each image feature point from the screenshot image as each first feature point, and to determine each image feature point from the standard image as each second feature point according to a preset feature extraction dimension, wherein the preset feature extraction dimension includes: a distribution of brightness values ​​corresponding to each pixel point contained in the image, intersections between each line contained in the image, and changes in pixel values ​​corresponding to each pixel point contained in the image;

[0109] A matching module 504, configured to obtain a matching result according to the similarity between each of the first feature points and each of the second feature points, wherein the matching result is used to indicate whether the image region corresponding to each of the first feature points in the screenshot image is an image region matching the standard image;

[0110] The execution module 505 is used to execute the task according to the matching result.

[0111] Optionally, the determination module 503 is specifically used to superimpose the standard image and the screenshot image several times, and use the image of the overlapping image area of ​​each superposition in the screenshot image as each sub-image obtained by superposition, wherein the overlapping image area of ​​each superposition is different; if it is determined that there is no sub-image in each sub-image whose similarity with the standard image exceeds a preset threshold, then according to a preset feature extraction dimension, each image feature point is determined from the screenshot image as each first feature point, and each image feature point is determined from the standard image as each second feature point.

[0112] Optionally, the execution module 505 is specifically used to, if it is determined that there is a sub-image in each sub-image whose similarity with the standard image exceeds a preset threshold, perform the task according to the position of the sub-image whose similarity with the standard image exceeds the preset threshold in the screenshot image.

[0113] Optionally, the execution module 505 is specifically used to, if it is determined based on the matching result that the image area corresponding to the first feature points in the screenshot image is not an image area that matches the standard image, then filter the first feature points according to the position of each first feature point in the screenshot image to obtain filtered first feature points; and execute the task according to the feature point matching results obtained based on the filtered first feature points.

[0114] Optionally, the execution module 505 is specifically used to determine, for each first feature point, other first feature points adjacent to the position of the first feature point in the screenshot image as reference first feature points corresponding to the first feature point; and to screen each first feature point according to the distance between each first feature point and the reference first feature point corresponding to each first feature point.

[0115] Optionally, the execution module 505 is specifically used to re-determine the screenshot image according to the image area corresponding to each filtered first feature point; determine each image feature point from the re-determined screenshot image as each third feature point according to a preset feature extraction dimension; and execute the task according to the matching results of each third feature point and each second feature point.

[0116] Optionally, the execution module 505 is specifically used to, if it is determined based on the matching result that the image area corresponding to each first feature point in the screenshot image is an image area that matches the standard image, then determine the position of the center point of the image area corresponding to each first feature point in the screenshot image based on the position of each first feature point in the screenshot image; identify the component element to be identified from the screenshot image based on the position of the center point, and perform a simulated touch operation on the identified component element to be identified; and perform a test task for the target application based on the interface displayed by the target application after responding to the simulated touch operation.

[0117] Optionally, the execution module 505 is specifically used to, if it is determined based on the matching result that the image area corresponding to the first feature points in the screenshot image is not an image area matching the standard image, then obtain an associated screenshot image corresponding to the screenshot image, the associated screenshot image and the screenshot image being interfaces displayed at different times during the operation of the target application; determine each image feature point from the associated screenshot image as each associated feature point according to a preset feature extraction dimension; and perform task execution based on the matching result of each associated feature point and each second feature point.

[0118] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 An image recognition method is provided.

[0119] This manual also provides Figure 6 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 6 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0120] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0121] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0122] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0123] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0124] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0125] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0134] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. An image recognition method, characterized in that: include: Capture the interface displayed when the target application is running and obtain a screenshot image; Determine at least one component element that needs to be identified from the interface based on the screenshot image as a component element to be identified, and obtain a standard image corresponding to the component element to be identified; According to a preset feature extraction dimension, each image feature point is determined from the screenshot image as each first feature point, and each image feature point is determined from the standard image as each second feature point, wherein the preset feature extraction dimension includes: a distribution of brightness values ​​corresponding to each pixel point contained in the image, intersections between each line contained in the image, and changes in pixel values ​​corresponding to each pixel point contained in the image; Obtaining a matching result according to the similarity between each of the first feature points and each of the second feature points, wherein the matching result is used to indicate whether an image region corresponding to each of the first feature points in the screenshot image is an image region that matches the standard image; The task is executed according to the matching result, wherein if it is determined according to the matching result that the image area corresponding to the first feature points in the screenshot image is not an image area that matches the standard image, the first feature points are filtered according to the position of each first feature point in the screenshot image to obtain the filtered first feature points, and the task is executed according to the feature point matching result obtained based on the filtered first feature points.

2. The method according to claim 1, characterized in that According to a preset feature extraction dimension, determining each image feature point from the screenshot image as each first feature point, and determining each image feature point from the standard image as each second feature point specifically includes: Superimposing the standard image and the screenshot image several times, and taking the image of the overlapping image area in the screenshot image of each superposition as each sub-image obtained by superposition, wherein the overlapping image area of ​​each superposition is different; If it is determined that there is no sub-image in the sub-images whose similarity with the standard image exceeds a preset threshold, then, according to the preset feature extraction dimension, each image feature point is determined from the screenshot image as each first feature point, and each image feature point is determined from the standard image as each second feature point.

3. The method according to claim 2, characterized in that The method further comprises: If it is determined that there is a sub-image among the sub-images whose similarity with the standard image exceeds a preset threshold, the task is executed according to the position of the sub-image whose similarity with the standard image exceeds the preset threshold in the screenshot image.

4. The method according to claim 1, characterized in that According to the position of each first feature point in the screenshot image, the first feature points are screened to obtain the screened first feature points, specifically including: For each first feature point, determine other first feature points adjacent to the first feature point in the screenshot image as reference first feature points corresponding to the first feature point; The first feature points are screened according to the distance between each first feature point and the reference first feature point corresponding to each first feature point.

5. The method according to claim 1, characterized in that According to the feature point matching results obtained based on the screened first feature points, the task is executed, specifically including: Re-determining the screenshot image according to the image area corresponding to each of the filtered first feature points; According to the preset feature extraction dimension, each image feature point is determined from the re-determined screenshot image as each third feature point; The task is executed according to the matching results of the third feature points and the second feature points.

6. The method according to claim 1, characterized in that According to the matching result, the task is executed, specifically including: If it is determined according to the matching result that the image area corresponding to each of the first feature points in the screenshot image is an image area that matches the standard image, then determining the position of the center point of the image area corresponding to each of the first feature points in the screenshot image according to the position of each of the first feature points in the screenshot image; According to the position of the center point, the component element to be identified is identified from the screenshot image, and a simulated touch operation is performed on the identified component element to be identified; A test task for the target application is performed according to an interface displayed by the target application after responding to the simulated touch operation.

7. The method according to claim 1, characterized in that According to the matching result, the task is executed, specifically including: If it is determined according to the matching result that the image area corresponding to each of the first feature points in the screenshot image is not an image area matching the standard image, then obtaining an associated screenshot image corresponding to the screenshot image, wherein the associated screenshot image and the screenshot image are interfaces displayed at different times during the running of the target application; According to a preset feature extraction dimension, determining each image feature point from the associated screenshot image as each associated feature point; The task is executed according to the matching results of the associated feature points and the second feature points.

8. An image recognition device, characterized in that: include: The capture module is used to capture the interface displayed when the target application is running and obtain a screenshot image; an acquisition module, used to determine at least one component element that needs to be identified from the interface based on the screenshot image as a component element to be identified, and acquire a standard image corresponding to the component element to be identified; a determination module, configured to determine, according to a preset feature extraction dimension, each image feature point from the screenshot image as each first feature point, and to determine each image feature point from the standard image as each second feature point, wherein the preset feature extraction dimension includes: a distribution of brightness values ​​corresponding to each pixel point contained in the image, intersections between each line contained in the image, and changes in pixel values ​​corresponding to each pixel point contained in the image; a matching module, configured to obtain a matching result according to the similarity between each of the first feature points and each of the second feature points, wherein the matching result is used to indicate whether the image area corresponding to each of the first feature points in the screenshot image is an image area matching the standard image; An execution module is used to execute a task according to the matching result, wherein if it is determined according to the matching result that the image area corresponding to each first feature point in the screenshot image is not an image area that matches the standard image, then the first feature points are screened according to the position of each first feature point in the screenshot image to obtain the screened first feature points, and the task is executed according to the feature point matching result obtained based on the screened first feature points.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images