Graphic verification code recognition method, device, electronic device and readable storage medium

Abstract

An embodiment of the present invention provides a method, apparatus, electronic device, and readable storage medium for recognizing graphical verification codes. Based on an object detection model trained by deep learning, the embodiment of the present invention can quickly extract the coordinate information of the outer frame of the icon to be clicked from the image of the area to be clicked, and then can extract each icon to be clicked. Then, according to the similarity, the icon to be clicked corresponding to each standard icon is determined, and according to the arrangement order of the standard icons, the icons to be clicked corresponding to each standard icon are clicked in sequence. The above recognition method avoids large-scale image annotation and classification, reduces the annotation workload, saves the implementation difficulty, and can be widely applied to many production scenarios where graphical click verification codes exist.

Description

Technical Field

[0001] The present invention relates to the technical field of verification code recognition, and in particular to a method, device, electronic device and readable storage medium for recognizing click-type graphic verification codes. Background Art

[0002] Currently, many websites use verification codes to verify whether it is a real manual operation during the access and query processes.

[0003] The graphic click verification code is one of the most common verification code forms. It is to click the corresponding icon position in the background image of the pattern to be clicked in the order of the icons in the legend to complete the verification code verification process.

[0004] Existing graphic click verification code recognition solutions need to detect and classify the images in the legend through computer vision algorithms and machine learning detection models, and at the same time detect and classify the area to be clicked. Then, according to the detection box position and category of the detection results of the legend part, compare the icon position and classification detected in the area to be clicked and click in order to complete the verification process.

[0005] However, the verification process of the above graphic click verification code has a large demand for data annotation required for classification, and since the verification code patterns are often updated in real time, and the algorithm or model needs to be updated synchronously when updating the icons in the icon library, the existing recognition methods for graphic click verification codes have poor practicability. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a method, device, electronic device and readable storage medium for recognizing graphic verification codes that solve the above problems or at least partially solve the above problems.

[0007] In a first aspect, an embodiment of the present invention provides a method for recognizing a graphic verification code, the method comprising:

[0008] Segmenting a legend image and an area image to be clicked from a verification code image;

[0009] Extracting each standard icon in the legend image and determining the arrangement order of each standard icon;

[0010] Inputting the area image to be clicked into a target detection model to output target coordinate information; the target detection model is used to determine the coordinate information of the image position where the outer frame of the icon to be clicked is located according to the area image to be clicked;

[0011] Cropping the icon to be clicked from the verification code image according to the target coordinate information;

[0012] Determine the icon to be clicked corresponding to each of the standard icons according to the similarity degree.

[0013] Click the icons to be clicked corresponding to each of the standard icons in sequence according to the arrangement order.

[0014] Optionally, in the graphic verification code recognition method, determining the icon to be clicked corresponding to each of the standard icons according to the similarity degree includes:

[0015] Extract the first feature vector of the standard icon and the second feature vector of the icon to be clicked.

[0016] For each of the standard icons, calculate the similarity degree between the first feature vector and the second feature vector.

[0017] Determine the icon to be clicked corresponding to the standard icon as the icon to be clicked to which the second feature vector with the highest similarity degree to the first feature vector belongs.

[0018] Optionally, in the graphic verification code recognition method, extracting the first feature vector of the standard icon and the second feature vector of the icon to be clicked includes:

[0019] Adopt a convolutional neural network with weight sharing to extract the features of the standard icon as a one-dimensional first feature vector and extract the features of the icon to be clicked as a one-dimensional second feature vector respectively.

[0020] Optionally, in the graphic verification code recognition method, extracting each standard icon in the legend image and determining the arrangement order of each of the standard icons includes:

[0021] Extract the legend image as a whole from the verification code image.

[0022] Perform pixel value scanning on the legend image column by column from left to right.

[0023] Record the image between the first column of pixels and the second column of pixels as a standard icon in sequence; the first column of pixels includes black pixels, and the previous column of pixels of the first column of pixels does not include black pixels; the second column of pixels is the column of pixels that is the first whole column of white pixels to the right of the first column of pixels.

[0024] Cut each of the standard icons from the legend image according to the recording result and sort them according to the cutting order.

[0025] Optionally, in the graphic verification code recognition method, calculating the similarity degree between the first feature vector and the second feature vector includes:

[0026] Calculate the similarity between the first feature vector and the second feature vector using cosine similarity.

[0027] Optionally, in the graphic verification code recognition method, the target detection model is pre-trained from the YOLOv5s model.

[0028] In a second aspect, an embodiment of the present invention provides a graphic verification code recognition device, which includes:

[0029] A segmentation module for segmenting a legend image and an image of the area to be clicked from the verification code image;

[0030] A first extraction module for extracting each standard icon in the legend image and determining the arrangement order of each standard icon;

[0031] A detection module for inputting the image of the area to be clicked into the target detection model and outputting target coordinate information; the target detection model is used to determine the coordinate information of the position of the outer frame of the icon to be clicked in the image according to the image of the area to be clicked;

[0032] A second extraction module for cropping the icon to be clicked from the verification code image according to the target coordinate information;

[0033] A matching module for determining the icon to be clicked corresponding to each standard icon according to the similarity;

[0034] A clicking module for sequentially clicking the icons to be clicked corresponding to each standard icon according to the arrangement order.

[0035] Optionally, in the device, the matching module includes:

[0036] A feature vector extraction unit for extracting the first feature vector of the standard icon and the second feature vector of the icon to be clicked;

[0037] A similarity calculation unit for calculating the similarity between the first feature vector and the second feature vector for each standard icon;

[0038] A determination unit for determining the icon to be clicked to which the second feature vector with the highest similarity to the first feature vector belongs as the icon to be clicked corresponding to the standard icon.

[0039] Optionally, in the device, the first extraction module includes:

[0040] A scanning unit for performing pixel value scanning on the legend image column by column from left to right;

[0041] A recording unit for sequentially recording the image between the first column of pixels and the second column of pixels as a standard icon; the first column of pixels includes black pixels, and the previous column of pixels of the first column of pixels does not include black pixels; the second column of pixels is the column of pixels located on the right side of the first column of pixels and the entire column is white pixels.

[0042] A cutting unit for cutting each of the standard icons from the legend image according to the recording result and sorting them according to the cutting order.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0044] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0045] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0046] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0047] In the embodiments of the present invention, a legend image and an image of the area to be clicked are segmented from the verification code image; each standard icon in the legend image is extracted, and the arrangement order of each standard icon is determined; the image of the area to be clicked is input into a target detection model to output target coordinate information; the target detection model is used to determine the coordinate information of the position of the outer frame of the icon to be clicked in the image of the area to be clicked according to the image of the area to be clicked; the icon to be clicked is cropped from the verification code image according to the target coordinate information; according to the similarity, the icon to be clicked corresponding to each legend image is determined; and according to the above arrangement order, the icons to be clicked corresponding to each standard icon are clicked in sequence. Because the target detection model trained based on deep learning can quickly extract the coordinate information of the outer frame of the icon to be clicked from the image of the area to be clicked, and then each icon to be clicked can be extracted. Then, according to the similarity, the icon to be clicked corresponding to each standard icon is determined, and according to the arrangement order of the standard icons, the icons to be clicked corresponding to each standard icon are clicked in sequence. The above recognition method avoids large-scale image annotation and classification, reduces the annotation workload, saves the implementation difficulty, and can be widely applied to many production scenarios where graphic click verification codes exist.

[0048] The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the content of the specification. And in order to make the above and other objects, features and advantages of the present invention more obvious and understandable, the following specifically illustrates the specific embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0050] Figure 1 The flowchart of the steps of an embodiment of a graphic verification code recognition method according to the present invention is shown;

[0051] Figure 2 The execution principle diagram according to an embodiment of the present invention is shown;

[0052] Figure 3 The block diagram of the structure of an embodiment of a graphic verification code recognition device according to the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0054] Referring to Figure 1 , the flowchart of the steps of an embodiment of a graphic verification code recognition method according to the present invention is shown, which may specifically include steps 101 to 106.

[0055] The embodiments of the present invention are applied to a terminal, and the terminal can be a mobile terminal or a non-mobile terminal, such as office devices such as mobile phones, tablets, computers, and laptops. The embodiments of the present invention are applicable to the recognition of graphic click verification codes.

[0056] Step 101: Segment the legend image and the image of the area to be clicked from the verification code image.

[0057] Among them, the above verification code image includes a legend image and an image of the area to be clicked. The above legend image is the overall image of the area where each standard icon for verification reference is located, and the above image of the area to be clicked is the overall image of the area where the icon to be clicked is located.

[0058] Among them, since the size of the verification code image is fixed and the position of the legend image in the verification code image is fixed, the area where the legend image is located can be accurately determined according to the absolute position of the verification code image and the relative position of the legend image relative to the verification code image, and the legend image can be segmented from the verification code image. The remaining part is the image of the area to be clicked, so as to obtain the above-mentioned legend image including each standard icon and the above-mentioned image of the area to be clicked.

[0059] Step 102: Extract each standard icon in the legend image and determine the arrangement order of each standard icon.

[0060] Among them, the position order of the above standard legends from left to right in the legend image, that is, the arrangement order of each standard legend. Therefore, after the legend image is segmented from the verification code image, the arrangement order of each standard icon can be determined according to the position of each standard legend in the legend image.

[0061] Step 103: Input the verification code image into the target detection model and output target coordinate information; the target detection model is used to determine the coordinate information of the position of the outer frame of the icon to be clicked in the image according to the image of the area to be clicked.

[0062] Among them, a high-precision target detection model for detecting the icon to be clicked that is different from the background of the area to be clicked is pre-trained by deep learning from the image of the area to be clicked in the verification code image. The detection result includes the coordinate information of the position of the outer frame of the icon to be clicked in the image, so as to facilitate subsequent cutting of each icon part to be clicked in the area to be clicked according to the detection result.

[0063] Optionally, the above target detection model is pre-trained by the deep learning method and obtained by training the YOLOv5s model in the model training manner.

[0064] Step 104: Extract the icon to be clicked from the verification code image according to the target coordinate information.

[0065] Among them, since the target coordinate information defines the position of the image to be clicked in the image of the area to be clicked, each icon to be clicked can be extracted from the verification code image according to the above target coordinate information, that is, the region of interest (ROI) of the icon is extracted using the detected outer frame of the icon.

[0066] Step 105: Determine the icon to be clicked corresponding to each standard icon according to the similarity.

[0067] Among them, for any one of the standard icons in each standard icon, through the metric learning method, calculate the similarity between each icon to be clicked and the standard icon, and determine the icon to be clicked with the highest similarity to the standard icon as its corresponding icon to be clicked.

[0068] Step 106: Click the icons to be clicked corresponding to each of the standard icons in sequence according to the arrangement order.

[0069] Among them, the above arrangement order is the order of each standard icon in the legend. Therefore, clicking the icons to be clicked corresponding to each standard icon according to the above arrangement order, that is, clicking the position of the icon with the highest similarity to the standard icon in the background image of the icon to be clicked, thus completing the verification code verification process. Specifically, according to the outer position of the corresponding icon frame in the area to be clicked, use methods such as RPA or Selenium for automated clicking, so as to complete the recognition and automated processing process of the entire graphic click type verification code.

[0070] In the embodiment of the present invention, because the object detection model trained based on deep learning can quickly extract the coordinate information of the outer frame of the icon to be clicked from the image of the area to be clicked, and then can extract each icon to be clicked, and then determine the icon to be clicked corresponding to each standard icon according to the similarity, and click the icons to be clicked corresponding to each standard icon in sequence according to the arrangement order of the standard icons. The above recognition method avoids large-scale image annotation and classification, reduces the annotation workload, saves the implementation difficulty and can be widely applied to many production scenarios where graphic click verification codes exist.

[0071] Optionally, in an implementation manner, the above step 102 specifically includes steps 201 to 203.

[0072] Step 201: Scan the pixel values of the legend image column by column from left to right.

[0073] Among them, the column scanning method scans the pixel values of the segmented legend image from left to right to determine the pixel situation of the entire column of pixels, and then determines whether it reaches the position where the outer frame of the corresponding standard icon is located.

[0074] Step 202: Record the image between the first column of pixels and the second column of pixels as a standard icon in sequence; the first column of pixels includes black pixels, and the previous column of pixels of the first column of pixels does not include black pixels; the second column of pixels is the first column of pixels located on the right side of the first column of pixels and whose entire column is white pixels.

[0075] Among them, when the first black pixel is encountered in the column pixels, recording starts. If an entire column is white pixels, recording stops and it is recorded as a legend icon. Then, continue to perform pixel value scanning on the part of the legend image that has not been scanned for pixel values column by column from left to right, and start recording when the first black pixel is encountered in the column pixels. If an entire column is white pixels, recording stops and it is recorded as a legend icon until the pixel value scanning of the entire legend image is completed.

[0076] Among them, since the first black pixel is the position of the left outer frame of the corresponding standard legend, and when an entire column is first encountered as white pixels, it is the position of the right outer frame of the standard legend. By the above method, the positions of each standard icon in the legend are determined, and then each standard icon in the legend icon is cut.

[0077] Step 203: Cut each of the standard icons from the legend image according to the recording result, and sort them according to the cutting order.

[0078] Among them, since the above recording result records the position of the left outer frame and the position of the right outer frame of the standard icon, so according to the above recording result, each standard icon in the legend image can be cut in the order from left to right; at the same time, because the cutting is from left to right, and the verification method of the graphic verification code is to click the icons to be clicked corresponding to each standard legend in the legend image in turn from left to right, so the standard icons can be directly sorted according to the order of cutting.

[0079] In this embodiment, the relevant principles of computer graphics are used to segment each standard icon of the legend image, and the standard icons can be quickly sorted directly according to the order of cutting, which is convenient for subsequent click verification of the icons.

[0080] Optionally, in one embodiment, the above step 105 includes steps 501 to 503.

[0081] Step 501: Extract the first feature vector of the standard icon and the second feature vector of the icon to be clicked.

[0082] Among them, the cut standard icons and the icons to be clicked are successively used with the same custom convolutional neural network (CNN) to extract feature vectors, so as to obtain the first feature vector of the above standard icon and the second feature vector of the icon to be clicked.

[0083] Optionally, in the above step 501, a convolutional neural network with weight sharing is used to extract the features of the standard icon as a one-dimensional first feature vector and extract the features of the icon to be clicked as a one-dimensional second feature vector. That is, the convolutional neural network that extracts the features from the standard icon as a one-dimensional vector and extracts the features from the icon to be clicked as a one-dimensional second feature vector is a siamese network. This not only ensures the consistency of the feature extraction method, reduces the adverse impact of the feature extraction method on similarity judgment, can quickly extract the feature vectors of each icon, but also greatly reduces the computational complexity in the subsequent similarity calculation process.

[0084] Step 502: For each of the standard icons, calculate the similarity between the first feature vector and the second feature vector.

[0085] Among them, the similarity between the feature vector of the standard icon and the feature vector of the icon to be clicked is calculated by means of metric learning. According to the order of the standard legends, each standard icon is sequentially matched with the icon to be clicked in the area to be clicked to determine the possibility of matching between each icon to be clicked and the standard icon.

[0086] Optionally, in the above step 502, the cosine similarity is used to calculate the similarity between the first feature vector and the second feature vector.

[0087] Step 503: Determine the icon to be clicked corresponding to the standard icon as the icon to be clicked to which the second feature vector with the highest similarity to the first feature vector belongs.

[0088] Among them, the icon to be clicked to which the second feature vector with the highest similarity to the first feature vector belongs is also the icon to be clicked with the highest similarity to the standard icon. Therefore, this icon to be clicked can be determined as the icon to be clicked corresponding to the standard icon.

[0089] In this embodiment, the cut standard legends and the icons to be clicked are respectively used with the same custom convolutional neural network (CNN) with weight sharing to extract features as one-dimensional vectors, and the cosine similarity is used to calculate the similarity between the standard icon and the icon to be clicked, so that the similarity matching between the standard icon and each icon to be clicked can be completed quickly and accurately.

[0090] Please refer to Figure 2 , which shows the execution principle diagram of the graphical verification code recognition method provided by the embodiment of the present invention.

[0091] Figure 2 As shown, in step 21, first, by means of the way of inputting by clicking on an icon, it is triggered to enter step 22 to perform image verification code recognition;

[0092] In step 22, according to the absolute position of the verification code image and in combination with the relative position of the legend image with respect to the verification code image, the area where the legend image is located is accurately determined, and then the legend image and the image of the area to be clicked are segmented from the verification code image;

[0093] In step 23, each standard icon of the legend image is segmented by using relevant principles of computer graphics to obtain each standard icon;

[0094] In step 24, for the image of the area to be clicked, a target detection model obtained by training with the YOLOv5s model is used to detect the pattern, detect the position of the icon to be clicked in the background image, and then enter step 25;

[0095] In step 25, the position of the detected icon frame is used to extract in the background image, and the icon ROI (region of interest) part is extracted;

[0096] In steps 26 and 27, the cut standard legend and the icon to be clicked respectively use the same custom convolutional neural network (CNN) with weight sharing to extract features as one-dimensional vectors;

[0097] In step 28, the cosine similarity is used to calculate the similarity between the feature vector of the standard icon and the feature vector of the icon to be clicked;

[0098] In step 29, the icon to be clicked corresponding to each standard icon is clicked in the above arrangement order of each standard icon, that is, the position of the icon in the background image of the pattern to be clicked with the highest similarity to the standard icon is clicked, thereby completing the entire recognition and automated processing process of the graphical click verification code.

[0099] In summary, the graphical verification code recognition method provided by the embodiments of the present invention, based on the target detection model trained by deep learning, can quickly extract the coordinate information of the outer frame of the icon to be clicked from the image of the area to be clicked, and then each icon to be clicked can be extracted. Then, according to the similarity, the icon to be clicked corresponding to each standard icon is determined, and according to the arrangement order of the standard icons, the icon to be clicked corresponding to each standard icon is clicked in turn. The above recognition method avoids large-scale image annotation and classification, reduces the annotation workload, saves the implementation difficulty, and can be widely applied to many production scenarios where graphical click verification codes exist.

[0100] Refer to Figure 3 , which shows a structural block diagram of an embodiment of a graphical verification code recognition device according to the present invention. The graphical verification code recognition device 30 includes:

[0101] A segmentation module 31, configured to segment the legend image and the image of the area to be clicked from the verification code image;

[0102] The first extraction module 32 is configured to extract each standard icon in the legend image and determine the arrangement order of each standard icon;

[0103] The detection module 33 is configured to input the image of the area to be clicked into a target detection model and output target coordinate information; the target detection model is used to determine the coordinate information of the position of the outer frame of the icon to be clicked in the image of the area to be clicked according to the image of the area to be clicked;

[0104] The second extraction module 34 is configured to extract the icon to be clicked from the verification code image according to the target coordinate information;

[0105] The matching module 35 is configured to determine the icon to be clicked corresponding to each standard icon according to the similarity;

[0106] The clicking module 36 is configured to click the icons to be clicked corresponding to each standard icon in sequence according to the arrangement order.

[0107] According to a graphic verification code recognition device of the present invention, because of the target detection model trained based on deep learning, the coordinate information of the outer frame of the icon to be clicked can be quickly extracted from the image of the area to be clicked, and then each icon to be clicked can be extracted. Then, according to the similarity, the icon to be clicked corresponding to each standard icon is determined, and according to the arrangement order of the standard icons, the icons to be clicked corresponding to each standard icon are clicked in sequence. The above recognition method avoids large-scale image annotation and classification, reduces the annotation workload, saves the implementation difficulty, and can be widely applied to many production scenarios where graphic click verification codes exist.

[0108] Optionally, in the device, the matching module 35 includes:

[0109] The feature vector extraction unit is configured to extract the first feature vector of the standard icon and the second feature vector of the icon to be clicked;

[0110] The similarity calculation unit is configured to calculate the similarity between the first feature vector and the second feature vector for each standard icon;

[0111] The determination unit is configured to determine the icon to be clicked to which the second feature vector with the highest similarity to the first feature vector belongs as the icon to be clicked corresponding to the standard icon.

[0112] Optionally, in the device, the first extraction module 32 includes:

[0113] The scanning unit is configured to perform pixel value scanning on the legend image column by column from left to right;

[0114] A recording unit for sequentially recording the image between the first column of pixels and the second column of pixels as a standard icon; the first column of pixels includes black pixels, and the previous column of pixels of the first column of pixels does not include black pixels; the second column of pixels is the column of pixels located on the right side of the first column of pixels and having the first entire column of white pixels.

[0115] A cutting unit for cutting each of the standard icons from the legend image according to the recording result and sorting them according to the cutting order.

[0116] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment.

[0117] Optionally, an embodiment of the present application further provides an electronic device, including a processor, a memory, a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements each process of the above-mentioned graphic verification code recognition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0118] It should be noted that the electronic device in the embodiment of the present application includes the above-mentioned mobile electronic device and non-mobile electronic device.

[0119] An embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-mentioned graphic verification code recognition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0120] Wherein, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0121] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement each process of the above-mentioned graphic verification code recognition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0122] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, system chip, chip system or system-on-chip, etc.

[0123] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used in accordance with this teaching. The structure required to construct such systems will be apparent from the above description. Additionally, the present invention is not directed to any particular programming language. It should be understood that the present invention as described herein may be implemented using various programming languages, and the description of a particular language above is for the purpose of disclosing the best mode of the present invention.

[0124] In the specification provided herein, numerous specific details are set forth. However, it can be understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0125] Similarly, it should be understood that, in order to streamline this disclosure and assist in understanding one or more of the various inventive aspects, in the foregoing description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, the inventive aspects lie in less than all of the features of the preceding single embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention.

[0126] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and disposed in one or more devices different from those of the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all the features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) can be replaced by an alternative feature that provides the same, equivalent, or similar purpose.

[0127] In addition, those skilled in the art can understand that although some of the embodiments described herein include certain features included in other embodiments rather than other features, the combination of features of different embodiments is meant to be within the scope of the present invention and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0128] Each component embodiment of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the file download device according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0129] It should be noted that the above embodiments illustrate the present invention rather than limit the present invention, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim listing several devices, several of these devices can be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names.

Claims

1. A method for recognizing graphic verification codes, characterized in that, The method includes: Segmenting a legend image and a to-be-clicked area image from a verification code image; Extracting each standard icon in the legend image and determining the arrangement order of each standard icon; Inputting the to-be-clicked area image into a target detection model to output target coordinate information; the target detection model is used to determine the coordinate information of the image position where the outer frame of the to-be-clicked icon is located according to the to-be-clicked area image; Cropping the to-be-clicked icon from the verification code image according to the target coordinate information; Determining the to-be-clicked icon corresponding to each standard icon according to the similarity; Sequentially clicking the to-be-clicked icons corresponding to each standard icon according to the arrangement order; Among them, extracting each standard icon in the legend image and determining the arrangement order of each standard icon includes: performing pixel value scanning on the legend image column by column from left to right; sequentially recording the image between the first column of pixels and the second column of pixels as a standard icon; the first column of pixels includes black pixels, and the previous column of pixels of the first column of pixels does not include black pixels; the second column of pixels is the column of pixels with the first entire column being white pixels on the right side of the first column of pixels; according to the recording result, cutting each standard icon from the legend image and sorting according to the cutting order.

2. The graphic verification code recognition method according to claim 1, wherein, Determining the to-be-clicked icon corresponding to each standard icon according to the similarity includes: Extracting a first feature vector of the standard icon and a second feature vector of the to-be-clicked icon; Calculating the similarity between the first feature vector and the second feature vector for each standard icon; Determining the to-be-clicked icon to which the second feature vector with the highest similarity to the first feature vector belongs as the to-be-clicked icon corresponding to the standard icon.

3. The graphic verification code recognition method according to claim 2, wherein Extracting the first feature vector of the standard icon and the second feature vector of the to-be-clicked icon includes: Using a convolutional neural network with weight sharing to extract the features of the standard icon as a one-dimensional first feature vector and extract the features of the to-be-clicked icon as a one-dimensional second feature vector respectively.

4. The graphic verification code recognition method according to claim 2, wherein Calculating the similarity between the first feature vector and the second feature vector includes: Calculating the similarity between the first feature vector and the second feature vector using cosine similarity.

5. The graphic verification code recognition method according to claim 1, wherein The target detection model is pre-trained by a YOLOv5s model.

6. A graphic verification code recognition device, characterized in that, The device includes: A segmentation module for segmenting a legend image and a to-be-clicked area image from a verification code image; A first extraction module for extracting each standard icon in the legend image and determining the arrangement order of each standard icon; A detection module for inputting the to-be-clicked area image into a target detection model to output target coordinate information; the target detection model is used to determine the coordinate information of the image position where the outer frame of the to-be-clicked icon is located according to the to-be-clicked area image; A second extraction module for cropping the to-be-clicked icon from the verification code image according to the target coordinate information; A matching module for determining the to-be-clicked icon corresponding to each standard icon according to the similarity; A click module, configured to click the to-be-clicked icons corresponding to the standard icons in sequence according to the arrangement order. The first extraction module includes: a scanning unit, configured to perform pixel value scanning on the legend image column by column from left to right; a recording unit, configured to record the image between the first column of pixels and the second column of pixels as a standard icon in sequence; the first column of pixels includes black pixels, and the previous column of pixels of the first column of pixels does not include black pixels; the second column of pixels is the column of pixels that is the first whole column of white pixels to the right of the first column of pixels; a cutting unit, configured to cut each standard icon from the legend image according to the recording result and sort them according to the cutting order.

7. The device according to claim 6, characterized in that, The matching module includes: A feature vector extraction unit, configured to extract a first feature vector of the standard icon and a second feature vector of the to-be-clicked icon. A similarity calculation unit, configured to calculate the similarity between the first feature vector and the second feature vector for each standard icon. A determination unit, configured to determine the to-be-clicked icon to which the second feature vector with the highest similarity to the first feature vector belongs as the to-be-clicked icon corresponding to the standard icon.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the graphic verification code recognition method according to any one of claims 1 to 5 are implemented.

9. A readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium. When the program or instruction is executed by the processor, the steps of the graphic verification code recognition method according to any one of claims 1 to 5 are implemented.

Images