Handwritten signature recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a handwritten signature recognition method and device, electronic equipment and storage medium. The handwritten signature recognition method comprises: obtaining a handwritten signature picture to be recognized; performing single character detection on the handwritten signature picture to obtain position information of a target character region; merging target character regions that need to be merged based on the position information to obtain a merged character region; recognizing characters corresponding to each character region, the character region including the target character region that has not been merged and the merged character region; comparing the characters with a user name in user information, and determining that a handwritten signature in the handwritten signature picture is valid when the comparison result is consistent. The handwritten signature recognition method can improve the accuracy of handwritten signature recognition.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a handwritten signature recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of internet technology, many offline businesses have gradually shifted to online services. In many scenarios, users are often required to provide a handwritten signature when conducting online transactions. The system then identifies and verifies the user's handwritten signature to ensure its validity.

[0003] In existing technologies, AI (Artificial Intelligence) is commonly used for handwritten signature recognition. A common approach is end-to-end text line recognition, which involves first detecting text lines and then directly performing end-to-end recognition on the entire text line. This method relies on semantic information and achieves excellent results in relatively well-structured printed text.

[0004] However, due to the large differences in character spacing and font in handwritten signatures, and the lack of semantic information, the above methods are not well applied to handwritten signature recognition scenarios, resulting in low accuracy in handwritten signature recognition. Summary of the Invention

[0005] In view of the above problems, this application proposes a handwritten signature recognition method, device, electronic device and storage medium to improve the accuracy of handwritten signature recognition.

[0006] According to one aspect of an embodiment of this application, a handwritten signature recognition method is provided, the method comprising:

[0007] Obtain the image of the handwritten signature to be recognized;

[0008] Perform individual character detection on the handwritten signature image to obtain the location information of the target text region;

[0009] Based on the location information, the target text regions that need to be merged are merged to obtain the merged text region;

[0010] Identify the text corresponding to each text region, wherein the text region includes the unmerged target text region and the merged text region;

[0011] The text is compared with the user's name in the user profile. If the comparison results are consistent, the handwritten signature in the handwritten signature image is determined to be valid.

[0012] Optionally, performing single-character detection on the handwritten signature image to obtain the location information of the target text region includes: using a pre-trained object detection model to perform single-character detection on the handwritten signature image to obtain the location information of the target text region; wherein, the object detection model is a model obtained by replacing the slice layer in the YOLO model with a convolutional layer.

[0013] Optionally, based on the location information, merging the target text regions to be merged to obtain a merged text region includes: calculating the lateral distance between adjacent target text regions based on the location information;

[0014] A merging threshold is calculated based on the horizontal distances mentioned above; adjacent target text regions with a horizontal distance less than or equal to the merging threshold are merged to obtain the merged text region.

[0015] Optionally, identifying the text corresponding to each text region includes: for each text region, using a pre-trained image classification model to obtain the confidence level of the text region belonging to each candidate text; and selecting at least one candidate text with the highest confidence level as the text corresponding to the text region.

[0016] Optionally, comparing the text with the user's name in the user profile includes: if the user's name is a single name, comparing the text with the single name; if the user's name is multiple names, concatenating the multiple names in different orders, and comparing the text with the concatenation result.

[0017] Optionally, the method further includes: when the comparison results are inconsistent: if the merged text region exists, the merged text region is split, and then the step of identifying the text corresponding to each text region is returned; if the merged text region does not exist, the handwritten signature in the handwritten signature image is determined to be invalid.

[0018] Optionally, splitting the merged text region includes: obtaining the lateral distance between adjacent target text regions contained in the merged text region; and splitting the adjacent target text region with the largest lateral distance into the target text region before merging.

[0019] According to another aspect of the embodiments of this application, a handwritten signature recognition device is provided, the device comprising:

[0020] The acquisition module is used to acquire images of the handwritten signature to be recognized.

[0021] The detection module is used to perform single-character detection on the handwritten signature image to obtain the location information of the target text region;

[0022] The merging module is used to merge the target text regions that need to be merged based on the location information to obtain a merged text region;

[0023] The recognition module is used to recognize the text corresponding to each text region, wherein the text region includes the unmerged target text region and the merged text region;

[0024] The comparison module is used to compare the text with the user's name in the user profile. If the comparison results are consistent, the handwritten signature in the handwritten signature image is determined to be valid.

[0025] Optionally, the detection module is specifically used to perform single-character detection on the handwritten signature image using a pre-trained target detection model to obtain the location information of the target character region; wherein, the target detection model is a model obtained by replacing the slice layer in the YOLO model with a convolutional layer.

[0026] Optionally, the merging module includes: a first calculation unit for calculating the lateral distance between adjacent target text regions based on the location information; a second calculation unit for calculating a merging threshold based on each of the lateral distances; and a region merging unit for merging adjacent target text regions whose lateral distance is less than or equal to the merging threshold to obtain the merged text region.

[0027] Optionally, the recognition module includes: a classification unit, used to obtain the confidence level of each text region belonging to each candidate text region using a pre-trained image classification model; and a selection unit, used to select at least one candidate text with the highest confidence level as the text region corresponding to the text region.

[0028] Optionally, the comparison module includes: a first comparison unit, configured to compare the text with the single name if the user name is a single name; and a second comparison unit, configured to compare the text with the spliced ​​result if the user name is multiple names, splicing the multiple names in different orders.

[0029] Optionally, the device further includes a processing module, configured to: when the comparison result of the comparison module is inconsistent, if the merged text region exists, split the merged text region and then call the recognition module to recognize the text corresponding to each text region; if the merged text region does not exist, determine that the handwritten signature in the handwritten signature image is invalid.

[0030] Optionally, the processing module includes: a distance acquisition unit, used to acquire the lateral distance between adjacent target text regions contained in the merged text region; and a splitting unit, used to split the adjacent target text region with the largest lateral distance into target text regions before merging.

[0031] According to another aspect of the embodiments of this application, an electronic device is provided, comprising: one or more processors; and one or more computer-readable storage media having instructions stored thereon; wherein, when the instructions are executed by the one or more processors, the processors perform the handwritten signature recognition method as described in any of the preceding claims.

[0032] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the handwritten signature recognition method as described in any of the preceding claims.

[0033] In this embodiment, the location information of the target text region is obtained by detecting individual characters in the handwritten signature image, enabling the localization of individual characters without relying on semantic information. By merging and cropping the target text region, the low recognition accuracy caused by differences in character spacing and font in handwritten signatures can be effectively overcome. By identifying the characters corresponding to each text region and comparing them with the user's name in the user profile to determine the recognition result, the problem of missed or incorrect character detection caused by differences in character spacing can be effectively overcome. Therefore, the handwritten signature recognition method of this embodiment can improve the accuracy of handwritten signature recognition. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some drawings of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of a handwritten signature with different character spacing according to an embodiment of this application.

[0036] Figure 2 This is a schematic diagram of handwritten text in a different font according to an embodiment of this application.

[0037] Figure 3 This is a flowchart illustrating the steps of a handwritten signature recognition method according to an embodiment of this application.

[0038] Figure 4 This is a schematic diagram of a handwritten signature with two names according to an embodiment of this application.

[0039] Figure 5 This is a flowchart illustrating the steps of another handwritten signature recognition method according to an embodiment of this application.

[0040] Figure 6 This is a schematic diagram illustrating one embodiment of obtaining a handwritten signature image according to this application.

[0041] Figure 7 This is a schematic diagram of a target text area according to an embodiment of this application.

[0042] Figure 8 This is a schematic diagram of target text region merging according to an embodiment of this application.

[0043] Figure 9 This is a schematic diagram of a text area according to an embodiment of this application.

[0044] Figure 10 This is a schematic diagram of the text corresponding to a text area in an embodiment of this application.

[0045] Figure 11 This is a schematic diagram of a handwritten signature recognition result according to an embodiment of this application.

[0046] Figure 12 This is a schematic diagram of another handwritten signature recognition result according to an embodiment of this application.

[0047] Figure 13 This is a schematic diagram of a handwritten signature recognition log in a non-transfer scenario according to an embodiment of this application.

[0048] Figure 14 This is a schematic diagram of a handwritten signature recognition log in a property transfer scenario according to an embodiment of this application.

[0049] Figure 15 This is a schematic diagram of a single-signature business registration form according to an embodiment of this application.

[0050] Figure 16 This is a schematic diagram of the recognition result of a single-signature business registration form according to an embodiment of this application.

[0051] Figure 17 This is a schematic diagram of another single-signature business registration form according to an embodiment of this application.

[0052] Figure 18 This is a schematic diagram of another single-signature business registration form recognition result according to an embodiment of this application.

[0053] Figure 19 This is a schematic diagram of a dual-signature business registration form according to an embodiment of this application.

[0054] Figure 20It is a schematic diagram of the recognition result of a double-signature business registration form according to an embodiment of the present application.

[0055] Figure 21 It is a schematic diagram of another double-signature business registration form according to an embodiment of the present application.

[0056] Figure 22 It is a schematic diagram of the recognition result of another double-signature business registration form according to an embodiment of the present application.

[0057] Figure 23 It is a structural block diagram of a handwritten signature recognition device according to an embodiment of the present application.

[0058] Figure 24 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0060] It should be noted that all actions of obtaining signals, information or data in the present application are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and with the authorization given by the owner of the corresponding device.

[0061] Since a handwritten signature is obtained by a user's handwriting, and the characters written by different users vary greatly, which is mainly reflected in the large difference in the character spacing of the handwritten signatures of different users, the large difference in the fonts written by different users, and so on. Due to the above-mentioned text differences, the existing end-to-end recognition method of text lines is prone to recognition errors.

[0062] Refer to Figure 1 , which shows a schematic diagram of a handwritten signature with different character spacings according to an embodiment of the present application. As Figure 1 shown, the character spacing in the handwritten signature "Li Dongzhi" is small, and the character spacing in the handwritten signature "Li Chaoqun" is large.

[0063] Refer to Figure 2 , which shows a schematic diagram of handwritten characters with different fonts according to an embodiment of the present application. As Figure 2 shown, for the character "苹", the fonts written by different users vary greatly.

[0064] To address the aforementioned issues, this application proposes a combination of target detection and image classification techniques. By treating each character as an independent target for detection and incorporating a merging and cropping algorithm, combined with high-precision character classification, the final recognition result is determined by matching user data from the system. This effectively overcomes problems such as missed or incorrect character detection caused by differences in character spacing. The following provides a detailed description of the handwritten signature recognition method according to this application.

[0065] The handwritten signature recognition method of this application can be applied to any business scenario that requires handwritten signatures, including but not limited to: communication services, banking services, contract signing services, etc. The handwritten signature recognition method can be executed by a business-related server or a business-related client.

[0066] Reference Figure 3 The diagram illustrates a flowchart of a handwritten signature recognition method according to an embodiment of this application.

[0067] like Figure 3 As shown, the handwritten signature recognition method may include the following steps:

[0068] Step 301: Obtain the image of the handwritten signature to be recognized.

[0069] When conducting online business, users typically sign a business electronic registration form with their handwritten signature. This handwritten signature is then stitched together with the form as an image. Therefore, it's possible to obtain the business electronic registration form with the handwritten signature and extract the image of the handwritten signature to be recognized from it. The image of the handwritten signature to be recognized contains the user's handwritten signature.

[0070] In practical applications, any applicable method can be used to extract the handwritten signature image to be identified from the electronic business registration form, depending on the actual situation.

[0071] For example, if the electronic business registration form is in PDF (Portable Document Format) format, and the device system performing the handwritten signature recognition method is based on the Python language, the PyMuPDF library can be used to extract the handwritten signature image to be recognized from the electronic business registration form, or the Pdf2Image library can be used to extract the handwritten signature image to be recognized from the electronic business registration form, and so on.

[0072] For example, if the electronic business registration form is in PDF format, and the device system performing the handwritten signature recognition method is based on the Java language, the iTextSharp tool can be used to extract the handwritten signature image to be recognized from the electronic business registration form, or the Photoshop tool can be used to extract the handwritten signature image to be recognized from the electronic business registration form, and so on.

[0073] Step 302: Perform single-character detection on the handwritten signature image to obtain the location information of the target text area.

[0074] The task of object detection is to identify all objects of interest in an image and determine their location and size. Therefore, object detection can be performed on individual characters in a handwritten signature image to be recognized, thereby obtaining the location information of target text regions in the handwritten signature image, where each target text region contains a single character.

[0075] In this embodiment of the application, when performing single character detection on the handwritten signature image to be identified, the target detection method may include, but is not limited to: R-CNN (Region Convolutional Neural Network) series models, SSD (Single Shot MultiBox Detector) series models, YOLO (You Only Look Once) series models, etc.

[0076] In this embodiment, considering that handwritten signature images mainly contain text, they are relatively simple and have high contrast. Therefore, it is unnecessary to use an overly complex object detection model to detect individual characters in the handwritten signature image. Thus, for relatively simple handwritten signature images, the existing object detection model structure can be simplified, and the simplified object detection model can be used to detect individual characters in the handwritten signature image.

[0077] For example, in this embodiment of the application, the YOLO model can be improved by replacing the Focus (slicing) layer in the YOLO model with a convolutional layer. Since the Focus layer in the YOLO model is mainly used to slice images to reduce their height and width, it is suitable for processing more complex images. However, for simpler handwritten signature images, there is no need to slice the handwritten signature image; feature extraction can be performed directly. Therefore, the Focus layer in the YOLO model can be removed, and a convolutional layer used for feature extraction can be used instead, thereby reducing FLOPS (computational cost).

[0078] The YOLO model can be YOLO v1, YOLO v2, YOLO v3, YOLO v5, etc. The convolutional layer can have any applicable dimension, such as 3×3, 6×6, etc.

[0079] The improved YOLO model (which replaces the focus layer in the YOLO model with a convolutional layer) is pre-trained.

[0080] First, obtain the sample handwritten signature image and its annotation information. The annotation information of the sample handwritten signature image represents the actual position information of each sample target text region in the sample handwritten signature image. Each sample target text region contains a single sample character, and its position information includes the center point coordinates, width, and height.

[0081] Next, an initial object detection model is constructed, which is the model obtained by replacing the Focus layer in the YOLO model with a convolutional layer. The parameters in the initial object detection model are initial parameters, and the parameters are iteratively optimized during model training.

[0082] Next, during the training process, the sample handwritten signature images are used as the input to the initial object detection model, and the annotation information of the sample handwritten signature images is used as the output target of the initial object detection model. The initial object detection model is then trained, and the trained model is used as the object detection model.

[0083] Specifically, the sample handwritten signature images are used as input to the initial object detection model, yielding its output, which is the predicted location information of each target text region in the sample handwritten signature images. Based on the output of the initial object detection model and the annotation information of the sample handwritten signature images, the model loss function can be calculated. Training is considered complete when the model loss function meets preset conditions (e.g., less than a certain threshold). The model loss function can include, but is not limited to, cross-entropy loss function, exponential loss function, squared loss function, absolute value loss function, etc.

[0084] When performing single-character detection on a handwritten signature image to be recognized, the aforementioned pre-trained target detection model can be used to perform single-character detection on the handwritten signature image to be recognized, and obtain the position information of each target character region in the handwritten signature image to be recognized.

[0085] Specifically, the handwritten signature image to be identified is used as input to the object detection model, and the output of the object detection model is the position information of each target text region in the handwritten signature image. Each target text region contains a single character, and the position information includes the center point coordinates (x, y), width w, and height h.

[0086] Step 303: Based on the position information, merge the target text regions that need to be merged to obtain a merged text region.

[0087] In the embodiments of the present application, considering that the character spacing differences written by different users are relatively large, and some users write in a scattered manner. For example, the left-right structure character "群" may be written in a scattered way, resulting in it being recognized as two characters "君" and "羊" during single character detection. In view of the above situation, in the embodiments of the present application, after obtaining the position information of the target text regions by performing single character detection on the handwritten signature picture, it can be determined whether there are target text regions that need to be merged, and the target text regions that need to be merged are merged to obtain a merged text region. Among them, the target text regions that need to be merged refer to the target text regions that belong to different parts of the same character.

[0088] Generally, in a handwritten signature, if the spacing between different parts of a certain character is relatively large, it may be recognized as different target text regions during the target detection process. However, the spacing between different parts of the same character during handwriting is usually smaller than the spacing between different characters. Therefore, the target text regions can be merged based on the horizontal distance between the target text regions.

[0089] In an optional implementation manner, the process of merging the target text regions that need to be merged based on the position information to obtain a merged text region may include the following steps A1 to A3:

[0090] Step A1: Based on the position information of the target text regions, calculate the horizontal distance between adjacent target text regions.

[0091] The position information of the target text region includes the center point coordinates (x, y) of the target text region. For adjacent target text regions, based on the center point coordinates (x, y) of the two target text regions in the adjacent target text regions, the horizontal distance between the adjacent target text regions can be calculated.

[0092] Specifically, the absolute value of the difference between the abscissas of the two target text regions in the adjacent target text regions is the horizontal distance between the adjacent target text regions.

[0093] For example, if target text region A and target text region B are adjacent target text regions, the abscissa of target text region A is represented as targetA.x, and the abscissa of target text region B is represented as targetB.x, then the horizontal distance S

[0094] , , AB , AB , ,

[0093] is expressed as follows:

[0094] S AB = abs(targetA.x - targetB.x)

[0095] Here, abs represents taking the absolute value.

[0096] Step A2: Calculate the merging threshold based on each of the horizontal distances.

[0097] For example, the average of all horizontal distances can be calculated as the merging threshold, or a weighted average of all horizontal distances can be calculated as the merging threshold, and so on.

[0098] For example, if the target text regions are A, B, C, D... in sequence, then the horizontal distances are S in sequence. AB S BC S CD ...If the merging threshold is the average of all horizontal spacings, then the merging threshold F is expressed as follows:

[0099] F = avg(S) AB +S BC +S CD +……)

[0100] Step A3: Merge adjacent target text regions whose horizontal distance is less than or equal to the merging threshold to obtain the merged text region.

[0101] The merged text region contains at least two target text regions.

[0102] For example, if S AB Less than or equal to ≤ F, S BC If >F, then target text region A and target text region B will be merged. If S AB ≤F,S BC If F ≤ F, then the target text region A, target text region B, and target text region C will be merged.

[0103] For example, a merge parameter `combine` can be set for each target text region. Target text regions that are merged with other target text regions can have their `combine` parameter set to "merge" (e.g., setting a value of 1 to indicate merge), while target text regions that are not merged with other target text regions can have their `combine` parameter set to "not merge" (e.g., setting a value of 0 to indicate not merge). A merge list `combine_List` can also be set to record the text regions to be merged and the target text regions to be merged.

[0104] Step 304: Identify the text corresponding to each text region.

[0105] After steps 303 and 304 above, the unmerged target text regions in the handwritten signature image and the merged text regions are obtained. The text within these regions can be considered as complete individual characters. By performing character recognition on each text region, the corresponding text can be obtained.

[0106] A text region is a single image containing a single character. Image classification refers to image processing methods that distinguish different categories of objects based on the different features they reflect in image information. It uses computers to quantitatively analyze images, classifying the image or each pixel or region within an image into one of several categories, replacing human visual interpretation. Therefore, image classification models can be used to identify the characters corresponding to each text region.

[0107] For example, image classification models may include, but are not limited to: ResNet (Residual Neural Network) series models, VGG (Visual Geometry Group Network) series models, Inception series models, Xception model, and so on.

[0108] The image classification model is trained in advance.

[0109] First, obtain the sample text image and its annotation information. The annotation information of the sample text image indicates the actual confidence level of the sample text image belonging to each candidate text. The sample text image is an image of a single character, and the candidate texts can be common characters in names, etc., with each candidate text corresponding to a text category.

[0110] Then, an initial image classification model is constructed, and the parameters in the initial object detection model are the initial parameters. The parameters are iteratively optimized during the model training process.

[0111] Next, during the training process, the sample text images are used as the input to the initial image classification model, and the annotation information of the sample text images is used as the output target of the initial image classification model. The initial image classification model is then trained, and the trained model is used as the image classification model.

[0112] Specifically, sample text images are used as input to the initial image classification model. Deep features of the sample text images are extracted from the initial image classification model, and the extracted features are analyzed to obtain the output of the initial image classification model. The output is the prediction confidence score of the sample text image belonging to each candidate character. Based on the output of the initial image classification model and the annotation information of the sample text images, the model loss function can be calculated. Training is considered complete when the model loss function meets preset conditions (e.g., less than a certain threshold). The model loss function can include, but is not limited to, cross-entropy loss function, exponential loss function, squared loss function, absolute value loss function, etc.

[0113] When identifying the text corresponding to each text region, for each text region, a pre-trained image classification model is used to obtain the confidence score of the text region belonging to each candidate text. Specifically, the image corresponding to the text region is used as the input of the image classification model. The deep features of the image corresponding to the text region are extracted in the image classification model. The extracted features are analyzed to obtain the output of the image classification model, which is the confidence score of the text region belonging to each candidate text.

[0114] At least one candidate character with the highest confidence level is selected as the character corresponding to the text region. Since the classification model may have errors in its calculation, multiple candidate characters with the highest confidence levels can be selected as the classification results, and then further comparisons can be made between these multiple candidate characters to improve the accuracy of recognition.

[0115] Step 305: Compare the text with the user's name in the user profile. If the comparison results are consistent, the handwritten signature in the handwritten signature image is determined to be valid.

[0116] The system pre-stores user information and retrieves the user's name from the user information. The user name consists of characters. The identified characters are then compared with the user's name in the user information. It can be understood that the order of the characters corresponding to the text area in the handwritten signature is the same as the order of the corresponding text area in the handwritten signature image. The comparison is a one-to-one correspondence; specifically, the first identified character is compared with the first character in the user's name, the second identified character with the second character in the user's name, and so on.

[0117] If the comparison results are consistent, the handwritten signature in the image to be identified is determined to be valid; if the comparison results are inconsistent, the handwritten signature in the image to be identified can be determined to be invalid, or other processing can be performed (described in the following embodiments).

[0118] In one possible scenario, the user's name in the user profile is a single name. In this case, the recognized text can be compared with the user's name in the user profile.

[0119] In another possible scenario, the user's name in the user profile may be multiple names. For example, in scenarios such as account transfers or user modifications in telecommunications services, the user's name may include both parties' signatures. Figure 4 This is a schematic diagram of a handwritten signature with two names, according to an embodiment of this application. Figure 4 As shown, the handwritten signature includes two names, "Li Chengcai" and "Li Qiusheng", so the corresponding user names in the user profile are also these two names.

[0120] In this situation, when signing by hand, the two names may be in different orders. If they are directly compared, the different orders may lead to recognition errors. Therefore, the multiple names can be concatenated in different orders, and the recognized text can be compared with the concatenated result.

[0121] For example, if two names are AAA and BBB, they can be concatenated into AAABBB and BBBAAA, and the recognized characters can be compared with AAABBB and BBBAAA respectively.

[0122] In this embodiment, the location information of the target text region is obtained by detecting individual characters in the handwritten signature image, enabling the localization of individual characters without relying on semantic information. By merging and cropping the target text region, the low recognition accuracy caused by differences in character spacing and font in handwritten signatures can be effectively overcome. By identifying the characters corresponding to each text region and comparing them with the user's name in the user profile to determine the recognition result, the problem of missed or incorrect character detection caused by differences in character spacing can be effectively overcome. Therefore, the handwritten signature recognition method of this embodiment can improve the accuracy of handwritten signature recognition.

[0123] Reference Figure 5 The flowchart illustrates the steps of another handwritten signature recognition method according to an embodiment of this application.

[0124] like Figure 5 As shown, the handwritten signature recognition method may include the following steps:

[0125] Step 501: Obtain the image of the handwritten signature to be recognized.

[0126] Step 502: Perform single-character detection on the handwritten signature image to obtain the location information of the target text area.

[0127] Step 503: Based on the location information, merge the target text regions that need to be merged to obtain the merged text region.

[0128] Step 504: Identify the text corresponding to each text region, wherein the text region includes the unmerged target text region and the merged text region.

[0129] Step 505: Compare the text with the user's name in the user profile.

[0130] Step 506: Determine if the comparison results are consistent. If yes, proceed to step 510; otherwise, proceed to step 507.

[0131] Step 507: Determine if there are any merged text regions. If yes, proceed to step 508; otherwise, proceed to step 509.

[0132] Step 508: Split the merged text region, then return to step 504.

[0133] If the identified text does not match the user's name in the user profile, and there are merged text regions, it can be considered whether target text regions that do not belong to the same text were mistakenly merged in step 503. In this case, the merged text regions can be split.

[0134] For example, the horizontal distance between adjacent target text regions contained in the merged text region can be obtained; the adjacent target text region with the largest horizontal distance is split into target text regions before merging. After splitting, the split merged target text regions can be deleted from the merge list combine_List, and the merge parameter combine of the split target text regions can be set to unmerged (for example, setting the value 0 indicates unmerged).

[0135] After splitting, return to step 504 and repeat steps 504 through 508. If the comparison result is still inconsistent after all target text regions have been split, the handwritten signature in the handwritten signature image is invalid. This method can solve the error problem caused by region merging and further improve the recognition accuracy.

[0136] Step 509: Determine that the handwritten signature in the handwritten signature image is invalid.

[0137] Step 510: Determine that the handwritten signature in the handwritten signature image is valid.

[0138] The following section will illustrate the handwritten signature recognition method with specific examples.

[0139] The handwritten signature recognition method of the embodiment of the present application includes the following processes:

[0140] 1. Obtain the handwritten signature picture from the business electronic registration form (PDF).

[0141] Figure 6 It is a schematic diagram of obtaining the handwritten signature picture in the embodiment of the present application. As Figure 1 shown, the business electronic registration form on the left contains the handwritten signature "Li Liqun", and the handwritten signature picture on the right containing the handwritten signature "Li Liqun" is extracted from it.

[0142] 2. Perform single-character detection on the handwritten signature picture to obtain the position information of the target text area.

[0143] Figure 7 It is a schematic diagram of a target text area in the embodiment of the present application. As Figure 7 shown, 5 target text areas are detected from the Figure 6 handwritten signature picture shown.

[0144] The target text area 1 is the area target1 where "wood" is located, with the abscissa x = 70, ordinate y = 81, width w = 86, and height h = 85;

[0145] The target text area 2 is the area target2 where "son" is located, with the abscissa x = 72, ordinate y = 163, width w = 103, and height h = 90;

[0146] The target text area 3 is the area target3 where "stand" is located, with the abscissa x = 223, ordinate y = 126, width w = 110, and height h = 105;

[0147] The target text area 4 is the area target4 where "lord" is located, with the abscissa x = 376, ordinate y = 123, width w = 77, and height h = 112;

[0148] The target text area 5 is the area target5 where "sheep" is located, with the abscissa x = 437, ordinate y = 122, width w = 73, and height h = 135.

[0149] 3. Based on the position information, merge the target text areas that need to be merged to obtain the merged text area.

[0150] Calculate the horizontal distance between each target text area, automatically set the merging threshold, and perform merging based on the merging threshold.

[0151] (1) The horizontal distance between the target text area 1 and the target text area 2 is:

[0152] target2.x - target1.x = 72 - 70 = 2

[0153] The horizontal distance between the target text area 2 and the target text area 3 is:

[0154] target3.x - target2.x = 223 - 72 = 151

[0155] The horizontal distance between the target text area 3 and the target text area 4 is:

[0156] target4.x - target3.x = 376 - 223 = 153

[0157] The horizontal distance between the target text area 4 and the target text area 5 is:

[0158] target5.x - target4.x = 437 - 376 = 61

[0159] (2) Take the average value of all horizontal distances as (2 + 151 + 153 + 61) / 4 = 91.75, and set it as the merging threshold.

[0160] (3) Merge the target text area 1 and the target text area 2 with horizontal distances less than the merging threshold, and merge the target text area 4 and the target text area 5.

[0161] At the same time, the merging parameter combine = 1 can also be set for the target text area 1 and the target text area 2, the target text area 4 and the target text area 5, and recorded in the merging list, and the merging parameter combine = 0 is set for other unmerged target text areas.

[0162] Figure 8 It is a schematic diagram of merging target text areas in an embodiment of the present application. As Figure 8 shown, the target text area 1 "wood" and the target text area 2 "son" with horizontal distances less than the merging threshold are merged, and the target text area 4 "jun" and the target text area 5 "sheep" are merged.

[0163] After merging, each independent text area is obtained. Figure 9 It is a schematic diagram of a text area in an embodiment of the present application. As Figure 9 shown, the text area 1 is the area where "Li" is located, and this text area is a merged text area; the text area 2 is the area where "Li" is located, and this text area is a target text area; the text area 3 is the area where "Qun" is located, and this text area is a merged text area.

[0164] 4. Identify the text corresponding to each text area.

[0165] Send the independent images corresponding to each text region into an image classification model, return the confidence levels of each text region belonging to each candidate text, and retain the text of the top 5 categories with the highest confidence levels as the recognition return parameter.

[0166] Figure 10 It is a schematic diagram of the text corresponding to a text region in an embodiment of this application. As Figure 10 shown, through text region 1, that is, the region where "Li" is located, 5 candidate texts "Li", "Ji", "Nai", "Xiang", "Xi" are recognized, and each candidate text has its corresponding confidence level; through text region 2, that is, the region where "Li" is located, 5 candidate texts "Li", "Zhi", "Dian", "Ying", "Wen" are recognized, and each candidate text has its corresponding confidence level; through text region 3, that is, the region where "Qun" is located, 5 candidate texts "Qun", "Pi", "Jun", "Jing", "Sui" are recognized, and each candidate text has its corresponding confidence level.

[0167] 5. Compare the recognized text with the user name in the user profile to obtain the handwritten signature recognition result.

[0168] Figure 11 It is a schematic diagram of a handwritten signature recognition result in an embodiment of this application. As Figure 11 shown, after the above processing, the handwritten signature recognition result can be displayed in a pop-up window, and the specifically displayed recognized text is: information such as Li Liqun.

[0169] In addition, if it is a double signature, such as transfer of ownership, modification of the user, etc., then first splice the two user names in different orders, and then compare them with the text obtained from the classification model respectively.

[0170] Figure 12 It is a schematic diagram of another handwritten signature recognition result in an embodiment of this application. As Figure 12 shown, it is a double-signature scenario. After the above processing, the handwritten signature recognition result can be displayed in a pop-up window, and the specifically displayed recognized text is: information such as Li Chengcai and Li Qiusheng.

[0171] The embodiment of this application can be applied to a business audit system and can generate a handwritten signature recognition log for the business audit system.

[0172] Figure 13 It is a schematic diagram of a handwritten signature recognition log for a non-transfer scenario in an embodiment of this application. The user name in the non-transfer scenario is a single name. Figure 13 In it, VALUE = 0 indicates consistent recognition, VALUE = -1 indicates inconsistent recognition, SECOND_VALUE indicates the signature of the recognition result, and CUST_NAME indicates the user name in the user profile.

[0173] Figure 14 This is a schematic diagram of a handwritten signature recognition log in a property transfer scenario according to an embodiment of this application. The user in the property transfer scenario uses two names. Figure 13 In this context, VALUE=0 indicates a consistent identification, VALUE=-1 indicates an inconsistent identification, SECOND_VALUE represents the signature of the identification result, and CUST_NAME represents the user's name in the user profile.

[0174] Figure 15 This is a schematic diagram of a single-signature business registration form according to an embodiment of this application. Figure 15 As shown, the user's name in the business registration form (that is, the customer's name in the figure) is Zhang Xudong, and the handwritten signature is Zhang Xudong.

[0175] Figure 16 This is a schematic diagram illustrating the recognition result of a single-signature business registration form according to an embodiment of this application. Figure 16 As shown, for Figure 15 In the identification results of the business registration form shown, VALUE=0, indicating that the identification is consistent.

[0176] Figure 17 This is a schematic diagram of another single-signature business registration form according to an embodiment of this application. Figure 17 As shown, the user's name in the business registration form (that is, the customer's name in the figure) is Zhao Zhengxian, and the handwritten signature is Ye Hailing.

[0177] Figure 18 This is a schematic diagram illustrating another single-signature business registration form recognition result according to an embodiment of this application. Figure 18 As shown, for Figure 17 In the identification result of the business registration form shown, VALUE = -1, indicating that the identification is inconsistent.

[0178] Figure 19 This is a schematic diagram of a dual-signature business registration form according to an embodiment of this application. Figure 19 As shown, the user names on the business registration form are Lai Wenbin and Liao Weihua, and the handwritten signatures are also Lai Wenbin and Liao Weihua.

[0179] Figure 20 This is a schematic diagram illustrating the recognition result of a dual-signature business registration form according to an embodiment of this application. Figure 20 As shown, for Figure 19 In the identification results of the business registration form shown, VALUE=0, indicating that the identification is consistent.

[0180] Figure 21 This is a schematic diagram of another dual-signature business registration form according to an embodiment of this application. Figure 21 As shown, the user names on the business registration form are Shu Chenglan and Kong Xianzhong, and the handwritten signatures are Shu Chenglan and Kong Zhong.

[0181] Figure 22 This is a schematic diagram illustrating another dual-signature business registration form recognition result according to an embodiment of this application. For example... Figure 22 As shown, for Figure 21 In the identification result of the business registration form shown, VALUE = -1, indicating that the identification is inconsistent.

[0182] In this embodiment, a handwritten signature recognition scheme combining object detection and image classification is adopted. The object detection and cropping algorithm identifies the target text regions of each character and crops them. The cropped image is then classified, and the classification results are compared with the customer names stored in the system to determine the final recognition result. Simultaneously, for dual-signature business scenarios such as property transfers, the names of the old and new users are concatenated in different orders and compared separately to ensure the accuracy of dual-signature recognition. Addressing the issue of misidentification in actual production, the existing model is improved. The image classification and matching algorithm enhances the efficiency and accuracy of handwritten signature recognition, saving significant manual auditing time and costs. This substantial improvement in the efficiency of electronic handwritten signature auditing ensures that business processing complies with regulatory requirements, greatly shortens processing time, and improves customer experience.

[0183] Reference Figure 23 The diagram shows a structural block diagram of a handwritten signature recognition device according to an embodiment of this application.

[0184] like Figure 23 As shown, a handwritten signature recognition device may include the following modules:

[0185] Module 2301 is used to acquire the handwritten signature image to be recognized;

[0186] Detection module 2302 is used to perform single character detection on the handwritten signature image to obtain the position information of the target text area;

[0187] The merging module 2303 is used to merge the target text regions to be merged into a merged text region based on the location information.

[0188] The recognition module 2304 is used to recognize the text corresponding to each text region, wherein the text region includes the unmerged target text region and the merged text region;

[0189] The comparison module 2305 is used to compare the text with the user's name in the user information. When the comparison result is consistent, the handwritten signature in the handwritten signature image is determined to be valid.

[0190] Optionally, the detection module 2302 is specifically used to perform single character detection on the handwritten signature image using a pre-trained target detection model to obtain the position information of the target character region; wherein, the target detection model is a model obtained by replacing the slice layer in the YOLO model with a convolutional layer.

[0191] Optionally, the merging module 2303 includes: a first calculation unit, used to calculate the lateral distance between adjacent target text regions based on the location information; a second calculation unit, used to calculate a merging threshold based on each of the lateral distances; and a region merging unit, used to merge adjacent target text regions whose lateral distance is less than or equal to the merging threshold to obtain the merged text region.

[0192] Optionally, the recognition module 2304 includes: a classification unit, used to obtain the confidence level of each text region belonging to each candidate text region using a pre-trained image classification model; and a selection unit, used to select at least one candidate text with the highest confidence level as the text region corresponding to the text region.

[0193] Optionally, the comparison module 2305 includes: a first comparison unit, configured to compare the text with the single name if the user name is a single name; and a second comparison unit, configured to compare the text with the spliced ​​result if the user name is multiple names, splicing the multiple names in different orders.

[0194] Optionally, the device further includes a processing module, configured to: when the comparison result of the comparison module is inconsistent, if the merged text region exists, split the merged text region and then call the recognition module to recognize the text corresponding to each text region; if the merged text region does not exist, determine that the handwritten signature in the handwritten signature image is invalid.

[0195] Optionally, the processing module includes: a distance acquisition unit, used to acquire the lateral distance between adjacent target text regions contained in the merged text region; and a splitting unit, used to split the adjacent target text region with the largest lateral distance into target text regions before merging.

[0196] In this embodiment, the location information of the target text region is obtained by detecting individual characters in the handwritten signature image, enabling the localization of individual characters without relying on semantic information. By merging and cropping the target text region, the low recognition accuracy caused by differences in character spacing and font in handwritten signatures can be effectively overcome. By identifying the characters corresponding to each text region and comparing them with the user's name in the user profile to determine the recognition result, the problem of missed or incorrect character detection caused by differences in character spacing can be effectively overcome. Therefore, the handwritten signature recognition method of this embodiment can improve the accuracy of handwritten signature recognition.

[0197] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0198] In embodiments of this application, an electronic device is also provided. This electronic device may include one or more processors and one or more computer-readable storage media storing instructions thereon, such as an application program. When the instructions are executed by the one or more processors, the processors cause the processors to perform the handwritten signature recognition method of any of the above embodiments.

[0199] Reference Figure 24 The diagram illustrates a schematic representation of an electronic device structure according to an embodiment of this application. Figure 24 As shown, the electronic device includes a processor 2401, a communication interface 2402, a memory 2403, and a communication bus 2404. The processor 2401, communication interface 2402, and memory 2403 communicate with each other via the communication bus 2404.

[0200] The memory 2403 is used to store computer programs.

[0201] When the processor 2401 executes the program stored in the memory 2403, it implements the handwritten signature recognition method of any of the above embodiments.

[0202] Communication interface 2402 is used for communication between the above-mentioned electronic device and other devices.

[0203] The aforementioned communication bus 2404 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, it is represented by only one thick line in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0204] The processor 2401 mentioned above may include, but is not limited to: a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0205] The aforementioned memory 2403 may include, but is not limited to: Read Only Memory (ROM), Random Access Memory (RAM), Compact Disc Read Only Memory (CD-ROM), Electronic Erasable Programmable Read Only Memory (EEPROM), Hard Disk, Floppy Disk, Flash Memory, etc.

[0206] In embodiments of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which can be executed by a processor of an electronic device, and when the computer program is executed by the processor, the processor performs the handwritten signature recognition method as described in any of the above embodiments.

[0207] The various embodiments in this specification are related to each other and are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0208] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0210] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0211] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0212] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0216] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0217] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. In summary, the content of this specification should not be construed as a limitation of this application.

Claims

1. A handwritten signature recognition method, characterized in that, The method includes: Obtain the image of the handwritten signature to be recognized; Perform individual character detection on the handwritten signature image to obtain the location information of the target text region; Based on the location information, the target text regions that need to be merged are merged to obtain the merged text region; Identify the text corresponding to each text region, wherein the text region includes the unmerged target text region and the merged text region; The text is compared with the user's name in the user profile. If the comparison results are consistent, the handwritten signature in the handwritten signature image is determined to be valid. When the comparison results are inconsistent: If the merged text region exists, the merged text region is split, and then the step of identifying the text corresponding to each text region is returned. If the merged text area does not exist, the handwritten signature in the handwritten signature image is determined to be invalid; The step of splitting the merged text region includes: Obtain the lateral distance between adjacent target text regions contained in the merged text region; The adjacent target text regions with the largest horizontal distance are split into the target text regions before merging.

2. The method according to claim 1, characterized in that, Perform single-character detection on the handwritten signature image to obtain the location information of the target text region, including: Using a pre-trained object detection model, single character detection is performed on the handwritten signature image to obtain the location information of the target character region; The target detection model is a model obtained by replacing the slice layers in the YOLO model with convolutional layers.

3. The method according to claim 1, characterized in that, Based on the location information, the target text regions to be merged are merged to obtain a merged text region, including: Calculate the lateral distance between adjacent target text regions based on the location information; Calculate the merging threshold based on the aforementioned lateral distances; The adjacent target text regions whose horizontal distance is less than or equal to the merging threshold are merged to obtain the merged text region.

4. The method according to claim 1, characterized in that, Identify the text corresponding to each text region, including: For each text region, a pre-trained image classification model is used to obtain the confidence level of the text region belonging to each candidate text. Select at least one candidate character with the highest confidence level as the character corresponding to the text region.

5. The method according to claim 1, characterized in that, The text is compared with the user's name in the user profile, including: If the user's name is a single name, then the text is compared with the single name; If the user name consists of multiple names, the multiple names are concatenated in different orders, and the text is compared with the concatenated result.

6. A handwritten signature recognition device, characterized in that, The device includes: The acquisition module is used to acquire images of the handwritten signature to be recognized. The detection module is used to perform single-character detection on the handwritten signature image to obtain the location information of the target text region; The merging module is used to merge the target text regions that need to be merged based on the location information to obtain a merged text region; The recognition module is used to recognize the text corresponding to each text region, wherein the text region includes the unmerged target text region and the merged text region; The comparison module is used to compare the text with the user's name in the user profile. When the comparison result is consistent, the handwritten signature in the handwritten signature image is determined to be valid. The processing module is configured to: when the comparison results of the comparison module are inconsistent, if the merged text region exists, split the merged text region and then call the recognition module to recognize the text corresponding to each text region; if the merged text region does not exist, determine that the handwritten signature in the handwritten signature image is invalid. The processing module includes: a distance acquisition unit, used to acquire the lateral distance between adjacent target text regions contained in the merged text region; and a splitting unit, used to split the adjacent target text region with the largest lateral distance into target text regions before merging.

7. An electronic device, characterized in that, include: One or more processors; and One or more computer-readable storage media on which instructions are stored; When the instruction is executed by the one or more processors, the processors perform the handwritten signature recognition method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, causes the processor to perform the handwritten signature recognition method as described in any one of claims 1 to 5.

Images