Verification code identification method, system and device based on twin network and medium

Abstract

The invention discloses a verification code identification method and system based on a twin network, a medium and equipment, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a verification code image and corresponding prompt information, and generating or extracting a target image sequence; extracting a plurality of candidate sub-images from the verification code image to form a candidate sub-image sequence and a position sequence; calculating the similarity between each target image in the target image sequence and each candidate sub-image in the candidate sub-image sequence by using a pre-trained twin network model, and constructing a similarity mapping table; constructing an ordered permutation combination to form a candidate click path set; calculating a path score of each path, and selecting an optimal click path; and according to the sequence of the candidate sub-graphs in the optimal click path, matching the positions corresponding to the candidate sub-graphs from the position sequence to form a screen click sequence. According to the scheme, verification code recognition is reconstructed from a classification problem to a matching problem, various changes of verification codes can be adapted, and the verification code recognition efficiency and practicability are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a verification code recognition method and system based on a twin network, a device and a medium. BACKGROUND

[0002] Verification code is a key barrier to Internet security. For protected enterprises, it is an important defense line against automated program malicious data crawling and preventing resource abuse. For data collection and analysis enterprises, it is a technical obstacle that needs to be overcome to achieve data automation collection. This attack and defense confrontation drives the continuous evolution of verification code technology. Its form has gradually evolved from simple distorted text and characters in the early stage to complex interactive verification methods such as sliding puzzles, text selection, icon selection, and spatial reasoning. The diversity and complexity continue to improve to more effectively distinguish between humans and machines.

[0003] For point selection type verification codes, the existing technical processing methods are divided into two categories: one is a step-by-step solution based on segmentation and recognition, which processes the recognition requirements of point selection type verification codes through step-by-step operations; the other is an integrated solution of end-to-end recognition, which completes the recognition task of point selection type verification codes with an integrated processing flow.

[0004] However, the existing technical solutions are essentially closed set classification problems, that is, the model can only recognize pre-defined categories in the training set and cannot handle subsequent new categories. Since the verification code technology is in a continuous attack and defense confrontation, the verification code is usually changed frequently, which leads to the need for repeated data collection and model retraining to adapt to new categories, seriously restricting the efficiency and practicality of verification code recognition. SUMMARY

[0005] The present application provides a verification code recognition method, system, device and medium based on a twin network, to solve the problem of low efficiency and practicality of existing verification code recognition methods.

[0006] In a first aspect, the present application provides a verification code recognition method based on a twin network, the method comprising: obtaining a verification code image and corresponding prompt information, generating or extracting a target image sequence T = [t1, t2, …, tn] according to the prompt information, n is the number of targets to be matched in the prompt information; n extracting a plurality of candidate subgraphs from the verification code image to form a candidate subgraph sequence C = [c1, c2, …, cm] and recording the position sequence R of the original position of each candidate subgraph in the verification code image, m is the number of candidate subgraphs; m extracting a plurality of candidate subgraphs from the verification code image to form a candidate subgraph sequence C = [c1, c2, …, cm] and recording the position sequence R of the original position of each candidate subgraph in the verification code image, m is the number of candidate subgraphs; ​using a pre-trained twin network model, calculating the similarity between each target image in the target image sequence T and each candidate subgraph in the candidate subgraph sequence C, and constructing a similarity mapping table S based on the similarity; Based on the similarity mapping table S, all ordered combinations of length n in the candidate subgraph sequence C are constructed to form a candidate click path set P, and each combination corresponds to a path of sequentially clicking candidate subgraphs; Calculate the path score of each path p in the candidate click path set P, and select the path with the highest path score as the optimal click path p best ; According to the order of the candidate subgraphs in the optimal click path p best , match the positions of the candidate subgraphs from the position sequence R to form a screen click sequence.

[0007] By adopting the above technical scheme, the verification code recognition is restructured from a classification problem to a matching problem, breaking through the limitation of traditional closed set classification model that can only recognize predefined categories, without the need to repeatedly collect data and retrain the model due to iterative updates of the verification code. The diversity of the verification code can be adaptively adapted, and a screen click sequence that meets the prompt requirements can be efficiently generated, significantly improving the efficiency and practicality of verification code recognition.

[0008] In a specific implementable scheme, the construction and training of the twin network model includes: A convolutional neural network with shared weights is used as a backbone feature extractor to extract semantic features from the input target image and candidate subgraph, respectively; The cross-attention mechanism is used to interact the feature sequences output by the backbone feature extractor. The feature sequence corresponding to the target image is used as the query vector, and the feature sequence corresponding to the candidate subgraph is used as the key vector and the value vector at the same time. The attention weight is calculated. The output after attention weighting is flattened into a two-dimensional vector, which is processed by a linear layer for dimension conversion, and a scalar value representing the similarity between the two images is output by the activation function.

[0009] By adopting the above technical scheme, based on the standard of unified feature extraction of the shared weight convolutional neural network, and combined with the cross-attention mechanism to strengthen the feature interaction between the target image and the candidate subgraph, and then processed by the linear layer for dimension conversion and the activation function, the similarity scalar value is accurately output. Effectively improve the effectiveness of image semantic feature extraction and the precision of similarity calculation, and enhance the robustness of the model to image differences such as noise and deformation.

[0010] In a specific implementable scheme, before using the cross-attention mechanism, it further includes: The three-dimensional feature tensor output by the backbone feature extractor is reshaped into a feature sequence. The dimensions of the feature sequence include batch size, sequence length, and feature dimension, so as to organize the spatial location information and channel information of the features into a sequence form that is adapted to attention calculation.

[0011] By adopting the above technical solution, the three-dimensional feature tensor output by the backbone feature extractor is reshaped into a feature sequence of a specific dimension. The spatial location information and channel information of the features are reasonably organized, so that the feature format is fully adapted to the computational requirements of the cross-attention mechanism, ensuring that the attention interaction can make full use of the feature information and improve the pertinence and effectiveness of the feature interaction.

[0012] In one specific feasible implementation, the training samples of the pre-trained Siamese network model are generated through data augmentation methods, including: Collect text and icons as basic templates, and apply at least one of the following perturbations to the basic templates: random affine transformation, noise injection, color dithering, or blurring, to simulate the morphological changes of real CAPTCHAs. Training sample pairs are constructed based on the perturbated base template, where two perturbated images from the same base template form a positive sample pair, and two perturbated images from different base templates form a negative sample pair, thus forming the training set of the Siamese network model.

[0013] By adopting the above technical solution, data augmentation technology is used to simulate the morphological changes of real CAPTCHAs. At the same time, positive and negative sample pairs are constructed to form a training set, which enriches the diversity and authenticity of the training samples, effectively improves the generalization ability of the Siamese network model, reduces the overfitting phenomenon of the model, and ensures the stable recognition performance of the model in real verification scenarios.

[0014] In a specific feasible implementation, multiple candidate sub-images are extracted from the CAPTCHA image using a pre-trained object detection model. The object detection model is used to detect and locate all clickable tiles in the CAPTCHA image.

[0015] By adopting the above technical solution, the pre-trained target detection model can accurately detect and locate all clickable blocks in the CAPTCHA image, ensuring the completeness and positional accuracy of the candidate sub-image extraction, avoiding omissions or mis-extraction of clickable targets, and providing a reliable foundation for the accurate matching of the target image and candidate sub-images in the future.

[0016] In one specific implementation, when the prompt message is text content, generating the target image sequence includes: Identify the font style features of each candidate sub-image in the CAPTCHA image; Based on the identified font style features, the text content is rendered into the corresponding target image.

[0017] By adopting the above technical solution, the target image corresponding to the text prompt is rendered according to the font style features of the candidate sub-image, eliminating the style domain difference between the text description and the image target, ensuring the consistency of the target image and the candidate sub-image in appearance style and semantic features, and further improving the accuracy of the similarity calculation between the two.

[0018] In one specific feasible implementation, the formula for calculating the path score includes: Where n represents the length of the target image sequence T, p[k] represents the k-th candidate sub-image in the click path, idx(p[k]) represents the index of p[k] in the candidate sub-image sequence C, and S(k,idx(p[k])) represents the similarity between the k-th target image in the target image sequence T and p[k], and the similarity is derived from the similarity mapping table S.

[0019] By adopting the above technical solution, the quality of the path is quantified by the sum of the similarity between the target image and the corresponding candidate sub-image. The optimal click path with the highest score can be accurately selected, ensuring that the final generated screen click sequence strictly matches the prompt information requirements, and effectively improving the success rate of CAPTCHA recognition.

[0020] A second aspect of this application provides a CAPTCHA recognition system based on a twin network, the system comprising: The extraction module is used to acquire the CAPTCHA image and the corresponding prompt information, generate or extract the target image sequence based on the prompt information, and extract multiple candidate sub-images from the CAPTCHA image to form a candidate sub-image sequence and a position sequence that records the original position of each candidate sub-image. The module is used to call the pre-trained Siamese network model, calculate the similarity between elements in the target image sequence and the candidate sub-image sequence, construct a similarity mapping table, form a set of candidate click paths, calculate the score of each path, and select the path with the highest score as the optimal click path. The output module is used to match the positions corresponding to each candidate sub-image from the position sequence according to the order of candidate sub-images in the optimal click path, and generate the screen click sequence.

[0021] A third aspect of this application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0022] A fourth aspect of this application provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0023] In summary, one or more technical solutions provided in this application embodiment have at least the following technical effects or advantages: 1. This solution calculates the similarity between the target and candidate subgraphs based on Siamese networks, and generates a click sequence by combining optimal path matching, eliminating the need to repeatedly collect data and retrain the model due to updates in CAPTCHA form and category. This design solves the problem of repeated training required by frequent iterations in traditional solutions, and can adaptively adapt to changes in point-selection CAPTCHAs, significantly reducing the response cost for CAPTCHA iteration.

[0024] 2. This scheme is based on Siamese networks and adopts shared weight convolutional neural networks and cross-attention mechanisms to unify feature extraction standards and enhance the feature interaction between the target and candidate subgraphs to capture semantic associations. It also uses data augmentation to generate diverse training samples to improve the model's ability to resist common interferences in real CAPTCHAs, such as noise, geometric deformation, and color jitter, and reduce the risk of misidentification. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a CAPTCHA recognition method based on a twin network provided in an embodiment of this application; Figure 2 This is a schematic diagram of a verification code image provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the extraction of candidate sub-graphs provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a twin network module provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating a method for matching a target image with candidate sub-images, as provided in an embodiment of this application. Figure 6 This is a schematic diagram of a verification code image clicking process provided in an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0027] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0028] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0029] Please refer to Figure 1 This paper presents a flowchart illustrating a CAPTCHA recognition method based on a twin network. This method can be implemented using a computer program, a microcontroller, or run on a CAPTCHA recognition system based on a twin network. The computer program can be integrated into a computer device or run as a standalone application. Specifically, the method includes steps S100 to S600, as follows: S100. Obtain the verification code image and corresponding prompt information, and generate or extract the target image sequence T = [t1, t2, ..., t] based on the prompt information. n ], where n is the number of targets to be matched in the prompt message; In some embodiments, the executing entity acquires the CAPTCHA image to be identified and the corresponding prompt information.

[0030] refer to Figure 2 In this embodiment of the application, the verification code image is an image containing independent tiles that the user needs to click and used to verify identity. The prompt information is information used to guide the user to click the tiles in the verification code image in a specific order.

[0031] The executing entity generates or extracts a target image sequence T based on the obtained prompt information. The target image sequence T is an ordered set of images corresponding to the target to be matched in the prompt information, and the expression for T is T = [t1, t2, ..., tt]. n ], where n is the number of targets to be matched in the prompt information, that is, the number of image elements contained in the target image sequence T.

[0032] Specifically, the executing entity determines the presentation form of the target to be matched in the prompt information. If the target to be matched exists in the form of an image, the executing entity directly extracts these images from the prompt information and arranges them in the order specified in the prompt information to form a target image sequence T. If the target to be matched exists in a non-image form, the executing entity converts these non-image-form target to be matched into images using image generation technology and then arranges them in the order specified in the prompt information to form a target image sequence T.

[0033] Based on the above embodiments, as another optional embodiment, when the prompt message is text content, generating the target image sequence includes: S101. Identify the font style features of each candidate sub-image in the CAPTCHA image; In some embodiments, when the prompt message is text content, the execution entity performs S101 to identify the font style features of the candidate sub-image.

[0034] In this embodiment of the application, the candidate sub-image is an independent image block that may be clicked and subsequently extracted from the CAPTCHA image; the font style features refer to the features of the text in the candidate sub-image, such as font type, font size, font color, font thickness, and font tilt angle, which can reflect the appearance style of the text.

[0035] Specifically, the executing entity preprocesses the candidate sub-images to be extracted from the CAPTCHA image, removes background noise from the candidate sub-images, highlights the text regions in the sub-images, and uses image feature recognition algorithms to analyze the text regions in the processed candidate sub-images, identify the font type, font size, font color, font thickness, and font tilt angle corresponding to the text region, and constitute the font style features of the candidate sub-images.

[0036] S102. Based on the identified font style features, render the text content into the corresponding target image.

[0037] In some embodiments, after completing the font style feature recognition in S101, the executing entity executes S102 to render the text content into a target image based on the feature.

[0038] Specifically, based on the font style characteristics of the candidate sub-images identified by S101, the executing entity determines the font parameters for rendering the text. These font parameters include a font type consistent with the candidate sub-image font, a font size matching the candidate sub-image font size, a color identical to the candidate sub-image font color, a font thickness matching the candidate sub-image font thickness, and a tilt angle similar to the candidate sub-image font tilt angle. The executing entity then calls an image rendering tool to render the text content in the prompt message according to the determined font parameters. Each text character corresponds to an independent image, and these independent images are arranged in the order of the text in the prompt message to form elements in the target image sequence T, thereby completing the construction of the target image sequence T.

[0039] S200. Extract multiple candidate sub-images from the CAPTCHA image to form a candidate sub-image sequence C = [c1, c2, ..., c m [and record the position sequence R of each candidate sub-image in the original position of the CAPTCHA image, where m is the number of candidate sub-images;] In some embodiments, after completing the construction of the target image sequence T in S100, the executing entity executes S200 to extract candidate sub-images from the CAPTCHA image, forming a candidate sub-image sequence C and a position sequence R.

[0040] refer to Figure 3 In this embodiment of the application, the candidate subgraph sequence C is an ordered set composed of all candidate subgraphs extracted from the CAPTCHA image in the order of extraction, and is expressed as C = [c1, c2, ..., cm], where m is the number of candidate subgraphs, that is, the number of candidate subgraphs contained in the candidate subgraph sequence C.

[0041] In this embodiment, the position sequence R is an ordered set that records the position information of each candidate sub-image in the original CAPTCHA image. This position information is used to subsequently determine the screen click coordinates.

[0042] Specifically, the executing entity uses object detection technology to scan the CAPTCHA image obtained in S100, detecting all regions in the image that may belong to the clickable tile. For each detected region, the executing entity extracts a corresponding image patch based on the region's boundary range; this image patch is the candidate sub-image. The executing entity arranges the extracted candidate sub-images sequentially according to the order in which the regions were detected, forming a candidate sub-image sequence C. Simultaneously, the executing entity records the position coordinates of each candidate sub-image in the original CAPTCHA image. These position coordinates are typically based on the top-left corner of the CAPTCHA image as the origin, recording the x-coordinate and y-coordinate of the top-left and bottom-right corner vertices of the candidate sub-image. The executing entity arranges the corresponding position coordinates sequentially according to the order in which the candidate sub-images are arranged in the candidate sub-image sequence C, forming a position sequence R.

[0043] Based on the above embodiments, as another optional embodiment, it includes: S201. Extract multiple candidate sub-images from the CAPTCHA image using a pre-trained object detection model. The object detection model is used to detect and locate all clickable tiles in the CAPTCHA image.

[0044] In some embodiments, the execution entity performs S201 to extract candidate subgraphs using a pre-trained object detection model, where the object detection module may be a YOLO model.

[0045] Specifically, the executing entity inputs the CAPTCHA image obtained in S100 into the pre-trained object detection model. The model analyzes the input CAPTCHA image, identifies the regions in the image that belong to the clickable tiles, and outputs the bounding box coordinates of each clickable tile. These bounding box coordinates include the x and y coordinates of the top-left and bottom-right vertices of the tile. Based on the bounding box coordinates output by the model, the executing entity extracts the corresponding tiles from the CAPTCHA image, i.e., candidate sub-images. The executing entity then constructs a candidate sub-image sequence C and a position sequence R based on the candidate sub-images.

[0046] S300. Using a pre-trained Siamese network model, calculate the similarity between each target image in the target image sequence T and each candidate sub-image in the candidate sub-image sequence C, and construct a similarity mapping table S based on the similarity. In some embodiments, after completing the construction of the target image sequence T in S100 and the candidate sub-image sequence C and position sequence R in S200, the executing entity executes S300 to calculate the similarity using a pre-trained Siamese network model and construct a similarity mapping table S.

[0047] refer to Figure 4 In this embodiment of the application, the Siamese network model refers to a deep learning model that has been trained to receive two image inputs and output a quantitative value that measures the similarity between the two images, i.e., similarity; the similarity mapping table S is a structured data set used to record the similarity between each target image in the target image sequence T and each candidate sub-image in the candidate sub-image sequence C, which can reflect the similarity relationship between different target images and different candidate sub-images.

[0048] Specifically, the executing entity calls a pre-trained Siamese network model, selects the first target image from the target image sequence T, and sequentially pairs this target image with each candidate sub-image in the candidate sub-image sequence C. These pairs are then input into the Siamese network model, which performs feature extraction and similarity calculation on each image pair, outputting the similarity between the target image and each candidate sub-image. The executing entity processes each target image in the target image sequence T in the same manner, obtaining the similarity between all target images and all candidate sub-images. Using the index of the target image in the target image sequence T and the index of the candidate sub-image in the candidate sub-image sequence C as identifiers, the executing entity records the corresponding similarities, constructing a similarity mapping table S.

[0049] Based on the above embodiments, as another optional embodiment, the construction and training of the Siamese network model includes: S301. A convolutional neural network with shared weights is used as the backbone feature extractor to extract semantic features from the input target image and candidate sub-images respectively. In this embodiment of the application, a convolutional neural network with shared weights refers to a convolutional neural network in which the branches used to process the target image and candidate sub-images use the same weight parameters; the backbone feature extractor is a module in the Siamese network model responsible for extracting features that can reflect the semantic information of the image from the input image.

[0050] Specifically, the execution entity selects a convolutional neural network as the backbone feature extractor of the Siamese network model and configures the convolutional neural network so that the branch processing the target image and the branch processing the candidate sub-image use the same weight parameters. That is, when the two branches perform feature extraction operations such as convolution and pooling, the convolution kernel parameters, pooling window sizes, etc. are kept consistent to ensure that the two branches have the same extraction criteria for image features, laying the foundation for accurate similarity calculation in the future.

[0051] In some embodiments, the selected convolutional neural network may be a series of residual network (ResNet) networks.

[0052] In some embodiments, the input to the Siamese network M is defined as X = (x t ,x c ), where x t x c The target image and candidate sub-images are represented by these symbols, respectively. Image preprocessing operations such as scaling and pixel value normalization are performed on the images. The scaled size is (105, 105), and the pixel values ​​are normalized to [0, 1]. A ResNet18 network is used as the backbone network with shared weights to extract x... t x c The image semantic features are obtained, and the output of the last convolutional layer is recorded as r(x). t ), r(x c The Reshap layer in a neural network is used to process r(x) respectively. t ), r(x c The dimensions are changed, and the output dimension parameter of the Reshap layer is (14, 512), that is, the output dimension is (b, 14, 512), where b represents the input annotation size.

[0053] S302. The feature sequences output by the backbone feature extractor are interacted using a cross-attention mechanism. The feature sequence corresponding to the target image is used as the query vector, and the feature sequences corresponding to the candidate sub-images are used as both key vectors and value vectors. Attention weights are also calculated. In this embodiment of the application, the cross-attention mechanism is a mechanism that enables information interaction between two different feature sequences and calculates attention weights based on the similarity between the sequences. The query vector is a feature sequence used to actively query information during the cross-attention calculation process. The key vector is a feature sequence used to match the query vector to calculate the attention weights. The value vector is a feature sequence of the feature sequence after interaction, which is weighted according to the calculated attention weights.

[0054] Specifically, the executing entity obtains the feature sequence output by the backbone feature extractor in S301 after processing the target image, and uses this feature sequence as the query vector in the cross-attention mechanism; simultaneously, it obtains the feature sequence output by the backbone feature extractor after processing the candidate sub-image, and uses this feature sequence as both the key vector and value vector in the cross-attention mechanism. The executing entity calculates the similarity between the query vector and each element in the key vector using the cross-attention mechanism, and determines the attention weight of each element based on this similarity. The attention weight reflects the importance of different elements in the key vector to the query vector. The executing entity then uses the calculated attention weights to perform a weighted summation of the value vectors to obtain the feature sequence after information interaction.

[0055] In some embodiments, a cross-attention mechanism is used to process r(x) t ), r(x c Attention learning is performed, where r(x) t As a Q-sequence, r(x) c As K and V sequences, attention weights are dynamically calculated, and the output is denoted as att(x). t ,x c ).

[0056] Based on the above embodiments, as another optional embodiment, before employing the cross-attention mechanism, the following is also included: S3021. Reshape the three-dimensional feature tensor output by the backbone feature extractor into a feature sequence. The dimensions of the feature sequence include batch size, sequence length, and feature dimension, so as to organize the spatial location information and channel information of the features into a sequence form suitable for attention calculation.

[0057] In this embodiment, the three-dimensional feature tensor is a feature data structure with height, width, and number of channels output by the backbone feature extractor; the feature sequence is a sequence data structure with three dimensions—batch size, sequence length, and feature dimension—obtained after transforming the three-dimensional feature tensor. Here, batch size refers to the number of image pairs input into the Siamese network model at one time, sequence length is the length of the three-dimensional feature tensor after its spatial dimension is expanded according to a specific rule, and feature dimension is the number of channels in the three-dimensional feature tensor.

[0058] In some embodiments, the execution entity performs a dimension reshaping operation on the three-dimensional feature tensor output by the backbone feature extractor, expanding the height and width dimensions of the three-dimensional feature tensor into a one-dimensional sequence dimension. This results in the reshaping of the feature data into a feature sequence, the dimensions of which include batch size, sequence length, and feature dimension. This transformation organizes the image spatial location information and channel feature information contained in the three-dimensional feature tensor into a sequence, thereby adapting to the processing requirements of the cross-attention mechanism for sequence-based data and ensuring that the subsequent cross-attention mechanism can effectively perform information interaction.

[0059] S303. Flatten the attention-weighted output into a two-dimensional vector, process it through a linear layer to perform dimensionality transformation, and output a scalar value representing the similarity between the two images through an activation function.

[0060] In some embodiments, the execution entity flattens the feature sequence after interaction via the cross-attention mechanism in S302, converting the multidimensional feature sequence into a two-dimensional vector. The first dimension of this two-dimensional vector is the batch size, and the second dimension is the number of feature elements. The execution entity inputs the flattened two-dimensional vector into a linear layer, which performs a linear transformation on the vector, adjusting the second dimension of the vector to 1, resulting in a one-dimensional vector. The execution entity then inputs the one-dimensional vector output from the linear layer into an activation function, which processes the one-dimensional vector and outputs a scalar value. This scalar value represents the similarity between the target image and the candidate sub-images input into the Siamese network model.

[0061] In some embodiments, the activation function selected can be the Sigmoid function, in which case the output scalar value ranges from 0 to 1. The closer the scalar value is to 1, the higher the similarity between the target image and the candidate sub-image; the closer the scalar value is to 0, the lower the similarity between the target image and the candidate sub-image.

[0062] In some embodiments, the Flatten layer in a neural network is used to apply att(x) t ,x c The dimension of att(x) is reduced to a two-dimensional vector, with an output dimension of (b, 512). Two linear layers are used to process the resulting att(x). t ,x c The similarity is processed and its dimensions are transformed. The first linear layer receives data in dimension (b, 512) and outputs data in dimension (b, 1024); the second linear layer receives data in dimension (b, 1024) and outputs data in dimension (b, 1). The output is activated using the Sigmoid function in the neural network, and the final similarity is recorded as s.

[0063] In some embodiments, the elements t in the target image sequence T are traversed sequentially. i The element c in the subgraph sequence Cj , forming matching pairs <t i ,c j > where i is the index in the target image sequence T, and j is the index in the sub-image sequence C. For example... Figure 5 As shown in the figure, each edge represents a matching pair, and there are a total of 9 matching pairs.

[0064] All matching pairs are calculated using a Siamese network model. <t i ,c j Similarity s ij The values ​​are recorded in a similarity mapping table S, where the keys are i and j, and the values ​​are s. ij . Figure 5 The specific similarity mapping table S is shown in Table 1: Table 1. Similarity mapping table S for CAPTCHA images nine (t1) dragon (t2) tea (t3) nine (c1) 0.98 0.63 0.02 Dragon (C2) 0.63 0.95 0.12 tea (c3) 0.02 0.12 0.95 Based on the above embodiments, as another optional embodiment, the training samples of the pre-trained Siamese network model are generated through data augmentation methods, including: S304. Collect text and icons as basic templates, and apply at least one of the following perturbations to the basic templates: random affine transformation, noise injection, color dithering, or blurring, to simulate the morphological changes of real CAPTCHAs. In some embodiments, the executing entity collects a large number of clear text and icon images, using these images as base templates. The executing entity processes each base template separately, applying at least one perturbation operation among random affine transformation, noise injection, color dithering, or blurring. For example, a text base template is first randomly rotated and Gaussian noise is injected to obtain a perturbated template image; another icon base template is subjected to color dithering and Gaussian blurring to obtain another perturbated template image. Through these operations, the executing entity obtains a large number of template images with different morphological changes to simulate the diverse forms of real CAPTCHAs.

[0065] S305. Construct training sample pairs based on the perturbated base template, where two perturbated images from the same base template form a positive sample pair, and two perturbated images from different base templates form a negative sample pair, thus forming the training set of the Siamese network model.

[0066] In this embodiment of the application, a positive sample pair refers to a sample pair consisting of two images with the same or highly similar semantic features, which is used to enable the model to learn the features of similar images; a negative sample pair refers to a sample pair consisting of two images with different semantic features, which is used to enable the model to learn the features of dissimilar images.

[0067] In some embodiments, BCELoss is used as the loss function and Adam as the optimizer. Text and icons are collected as templates, and image processing techniques are used to simulate the generation of a large number of training sample pairs X, where x in sample pair X... t ,x c Models from the same template are labeled 1; otherwise, they are labeled 0. Iterative training is performed until the model converges, and the optimal model M is saved. b .

[0068] Specifically, the execution entity selects samples from the perturbed template images obtained in S304. If two perturbed template images originate from the same base template, the execution entity will combine these two images into a positive sample pair and set label 1 for the positive sample pair. Label 1 is used to identify that the two images in the sample pair are similar.

[0069] If the two perturbated template images come from different base templates, the execution entity will combine the two images into a negative sample pair and set label 0 for the negative sample pair. Label 0 is used to identify that the two images in the sample pair are not similar.

[0070] The executing entity constructs a large number of positive and negative sample pairs in the same manner, and then organizes all the constructed positive and negative sample pairs together to form a training set for training the Siamese network model. The executing entity uses this training set to iteratively train the Siamese network model, adjusting the model parameters to reduce the model's prediction error, until the model's loss value on the validation set remains stable for multiple consecutive rounds.

[0071] S400. Based on the similarity mapping table S, construct all ordered permutations and combinations of length n in the candidate subgraph sequence C to form a candidate click path set P, where each permutation and combination corresponds to a path that clicks on the candidate subgraphs in sequence. In this embodiment, ordered permutation and combination refers to selecting n candidate sub-images from the candidate sub-image sequence C and arranging them in a specific order to form a combination. The candidate click path set P is a set of click paths corresponding to all possible ordered permutations and combinations. A click path is a sequence representing the sequential clicking of candidate sub-images in a specific order, reflecting the order of clicks. The executing entity first determines the number of candidate sub-images to be selected from the candidate sub-image sequence C, which is consistent with the length n of the target image sequence T, to ensure that the number of candidate sub-images in the click path is the same as the number of targets to be matched. Then, the executing entity uses a permutation and combination algorithm to select n candidate sub-images from the candidate sub-image sequence C, and performs all possible ordered permutations on the selected n candidate sub-images. Each ordered permutation corresponds to a click path, and the order of the click paths is the order of clicking the candidate sub-images.

[0072] In some embodiments, three elements are arbitrarily selected from C to form permutations and combinations, where any permutation is considered a click path p = [c idx1 ,c idx2 ,c idx3 Let p be an ordered set of elements representing the positions clicked sequentially, and idx be the index of an element in c. For example, p1 = [c1, c2, c3] represents clicking on candidate targets such as "nine", "dragon", and "tea" in the image sequentially. All possible paths p are grouped into a set, denoted as P = {p1, p2, ..., p...}. z}, where z = A m n =6, which is the number of elements in set P.

[0073] S500. Calculate the path score of each path p in the candidate click path set P, and select the path with the highest path score as the optimal click path p. best ; In this embodiment, the path score refers to a quantitative value calculated based on the similarity mapping table S, used to measure the quality of a click path. A higher path score indicates a higher overall similarity between the candidate sub-images and the corresponding target image within the click path; the optimal click path p best It refers to the click path with the highest path score in the candidate click path set P, and the click path that best meets the requirements of the prompt message.

[0074] Based on the above embodiments, as another optional embodiment, the formula for calculating the path score includes: Where n represents the length of the target image sequence T, p[k] represents the k-th candidate sub-image in the click path, idx(p[k]) represents the index of p[k] in the candidate sub-image sequence C, and S(k,idx(p[k])) represents the similarity between the k-th target image in the target image sequence T and p[k], and the similarity is derived from the similarity mapping table S.

[0075] Specifically, the executing entity selects the first click path p from the candidate click path set P, and calculates the path score of the path according to the similarity mapping table S. The specific calculation process includes: selecting the first candidate sub-image from the click path p, determining the index of the candidate sub-image in the candidate sub-image sequence C, and simultaneously determining the index of the first target image in the target image sequence T, and finding the similarity between the two indices from the similarity mapping table S; sequentially finding the similarity between each candidate sub-image in the click path p and the corresponding target image in the target image sequence T, and summing these similarities to obtain the path score of the click path p.

[0076] Furthermore, the executing entity calculates the path score for each click path in the candidate click path set P. After calculating the path scores for all click paths, the executing entity compares all path scores, identifies the path with the highest score, and determines this path as the optimal click path p. best .

[0077] For example, for Figure 5 In the example, p1 = [c1, c2, c3], representing the click order as "nine", "dragon", "tea", score(p k ) = 2.88; p2 = [c2,c1,c3], representing the click order as "Dragon", "Nine", "Tea", score(p k The score is 2.21. The optimal path is p1 = [c1, c2, c3], with a score of 2.88.

[0078] S600, Based on the optimal click path p best The order of candidate subgraphs is determined by matching the corresponding positions of each candidate subgraph from the position sequence R to form a screen click sequence.

[0079] In this embodiment of the application, the screen click sequence refers to the sequence following the optimal click path p. best The sequence of coordinates of each candidate sub-image on the screen is recorded in order, and this sequence can be directly used to control the device to perform click operations on the screen.

[0080] In some embodiments, the executing entity starts from the optimal click path p best The first candidate sub-image is selected, and its index in the candidate sub-image sequence C is determined. Based on this index, the corresponding position coordinates are found in the position sequence R. These position coordinates are the coordinates of the candidate sub-image in the original CAPTCHA image. The optimal click path p is then found sequentially. best For each candidate sub-image, find its corresponding position coordinates in the position sequence R, and then sort these position coordinates according to the optimal click path p. best The candidate sub-images are arranged in order to form a screen click sequence.

[0081] refer to Figure 6 Optimal click path p best Given p1 = [c1, c2, c3], click in the order of "Nine", "Dragon", and "Tea".

[0082] In some embodiments, the executing entity can control an input device, such as a mouse or touchscreen, to click sequentially at corresponding positions on the screen according to the screen click sequence, thereby completing the recognition and verification of the verification code.

[0083] Based on the above embodiments, as another optional embodiment, this application provides a CAPTCHA recognition system based on a twin network, the system comprising: The extraction module is used to acquire the CAPTCHA image and the corresponding prompt information, generate or extract the target image sequence based on the prompt information, and extract multiple candidate sub-images from the CAPTCHA image to form a candidate sub-image sequence and a position sequence that records the original position of each candidate sub-image. The module is used to call the pre-trained Siamese network model, calculate the similarity between elements in the target image sequence and the candidate sub-image sequence, construct a similarity mapping table, form a set of candidate click paths, calculate the score of each path, and select the path with the highest score as the optimal click path. The output module is used to match the positions corresponding to each candidate sub-image from the position sequence according to the order of candidate sub-images in the optimal click path, and generate the screen click sequence.

[0084] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0085] Based on the above embodiments, as another optional embodiment, the present application embodiment may further include a computer storage medium, which may store multiple instructions adapted for loading by a processor and executing a method of the above embodiments. For the specific execution process, please refer to the detailed description of the above embodiments, which will not be repeated here.

[0086] Based on the above embodiments, as another optional embodiment, this application embodiment may further include an electronic device. The electronic device may include: at least one processor, at least one communication bus, a user interface, at least one network interface, and a memory.

[0087] The communication bus is used to enable communication between these components.

[0088] The user interface may include a display screen and a camera. Optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0089] The network interface may include standard wired interfaces and wireless interfaces (such as Wi-Fi interfaces).

[0090] The processor may include one or more processing cores. It connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0091] The memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. As a computer storage medium, the memory may include an operating system, a network communication module, a user interface module, and an application program of one method.

[0092] In electronic devices, the user interface is primarily used to provide an input interface for users and to acquire user input data; while the processor can be used to call an application program stored in memory that represents a method. When executed by one or more processors, this causes the electronic device to perform one or more methods as described in the above embodiments. It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0093] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0094] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0098] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0099] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for recognizing a CAPTCHA based on a twin network, characterized in that, The method comprises: Acquire the verification code image and the corresponding prompt information, generate or extract the target image sequence T = [t1, t2, …, t n ] according to the prompt information, n is the number of targets to be matched in the prompt information; Multiple candidate sub-images are extracted from the CAPTCHA image to form a candidate sub-image sequence C = [c1, c2, ..., c3]. m [and record the position sequence R of each candidate sub-image in the original position of the CAPTCHA image, where m is the number of candidate sub-images;] Using a pre-trained twin network model, the similarity between each target image in the target image sequence T and each candidate sub-image in the candidate sub-image sequence C is calculated, and a similarity mapping table S is constructed based on the similarity; Based on the similarity mapping table S, all ordered arrangement combinations of length n in the candidate sub-image sequence C are constructed to form a candidate click path set P, and each arrangement combination corresponds to a path of sequentially clicking candidate sub-images. calculating a path score of each path p in the candidate click path set P, and selecting a path with the highest path score as the optimal click path p best ; According to the optimal click path p best The order of the candidate subgraphs, match the positions corresponding to each of the candidate subgraphs from the position sequence R, to form a screen click sequence. 2.The twin network based CAPTCHA recognition method of claim 1, wherein, The construction and training of the twin network model comprises: A convolutional neural network with shared weights is used as a backbone feature extractor to extract semantic features of the input target image and candidate sub-image respectively; The cross-attention mechanism is used to interact with the feature sequence output by the backbone feature extractor, the feature sequence corresponding to the target image is used as the query vector, the feature sequence corresponding to the candidate sub-image is used as the key vector and the value vector at the same time, and the attention weight is calculated; The output after attention weighting is flattened into a two-dimensional vector, which is processed by a linear layer for dimension conversion, and a scalar value representing the similarity between the two images is output through an activation function. 3.The twin network based CAPTCHA recognition method of claim 2, wherein, Before using the cross-attention mechanism, it also includes: The three-dimensional feature tensor output by the backbone feature extractor is reshaped into a feature sequence, and the dimensions of the feature sequence include batch size, sequence length, and feature dimension, so as to organize the spatial position information and channel information of the feature into a sequence form suitable for attention calculation. 4.The twin network based CAPTCHA recognition method of claim 1, wherein, The training samples of the pre-trained twin network model are generated by a data enhancement method, which comprises: Collecting text and icons as basic templates, applying at least one disturbance such as random affine transformation, noise injection, color jittering or blur processing to the basic templates to simulate the morphological changes of real verification codes; Based on the disturbed basic templates, a training sample pair is constructed, wherein two disturbed images from the same basic template form a positive sample pair, and two disturbed images from different basic templates form a negative sample pair, forming a training set of the twin network model. 5.The twin network based CAPTCHA recognition method of claim 1, wherein, A plurality of candidate sub-images are extracted from the verification code image by a pre-trained target detection model, and the target detection model is used to detect and locate all the sub-images to be clicked in the verification code image. 6.The twin network based CAPTCHA recognition method of claim 1, wherein, When the prompt information is text content, the generation of the target image sequence comprises: Identifying the font style features of each candidate sub-image in the verification code image; Based on the identified font style features, the text content is rendered into corresponding target images. 7.The twin network based CAPTCHA recognition method of claim 1, wherein, The calculation formula of the path score comprises: Wherein, n represents the length of the target image sequence T, p[k] represents the kth candidate sub-image in the click path, idx(p[k]) represents the index of p[k] in the candidate sub-image sequence C, S(k, idx(p[k])) represents the similarity between the kth target image in the target image sequence T and p[k], and the similarity is derived from the similarity mapping table S. 8.A system for recognizing a CAPTCHA based on a twin network, characterized in that, The system comprises: The extraction module is configured to acquire a verification code image and corresponding prompt information, generate or extract a target image sequence according to the prompt information, and extract a plurality of candidate sub-images from the verification code image to form a candidate sub-image sequence and a position sequence recording original positions of the candidate sub-images; The construction module is configured to call a pre-trained twin network model, calculate similarities between the target image sequence and elements in the candidate sub-image sequence, construct a similarity mapping table, form a candidate click path set, calculate scores of each path, and select a path with the highest score as an optimal click path; The output module is configured to match positions corresponding to the candidate sub-images in the optimal click path from the position sequence to generate a screen click sequence.

9. An electronic device, comprising: The electronic device includes a processor, a memory, a user interface, and a network interface, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded and executed by a processor to perform the method of any one of claims 1-7.

Images