Image processing method and device based on key point detection, equipment and medium

Abstract

The invention discloses an image processing method and device based on key point detection, equipment and a medium, key point data of to-be-recognized image data is extracted based on a pre-trained ViTPose model, so that the coordinate range of each column in the to-be-recognized image data is calibrated, more accurate content recognition and positioning can be provided for an OCR recognition module, and the recognition efficiency of the to-be-recognized image data is improved. The pre-trained ViTPose model can capture global context information by relying on a self-attention mechanism, key points in image data can be detected more accurately for a possible long-distance dependency relationship between different key points, and a content recognition result corresponding to each column is finally obtained. The technical problems that when the certificate format is changed or a non-standard text exists, recognition errors are likely to occur when the OCR technology is singly used for recognizing the picture, the robustness of post-processing is insufficient, and information extraction is likely to be inaccurate or missed are solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, equipment and medium based on key point detection. Background Art

[0002] In today's globalized economic environment, letters of credit, as an important international trade payment tool, play a vital role in protecting the rights and interests of both parties and promoting the smooth progress of transactions. The terms and conditions of a letter of credit specify in detail all aspects of the transaction, including but not limited to the description of the goods, the amount, the delivery period, the mode of transportation, the payment terms, etc. Therefore, accurately and efficiently extracting the terms of the letter of credit is a key task for banks, importers and exporters, and related financial institutions.

[0003] However, the traditional method of extracting letter of credit terms faces some challenges, mainly including:

[0004] Instability of OCR recognition: Traditional OCR (optical character recognition) technology may make recognition errors when processing letters of credit with poor printing quality, handwritten text or special formats. These errors will directly affect the accuracy of subsequent clause extraction, thereby affecting the efficiency and reliability of the entire business process.

[0005] Limitations of post-processing: Traditional OCR (optical character recognition) technology usually requires complex post-processing steps to extract key information after recognizing the letter of credit text. These post-processing steps often rely on preset rules and patterns. When the format of the letter of credit changes or there is non-standard text, the post-processing is not robust enough, which may lead to inaccurate or missing information extraction. Summary of the invention

[0006] In order to solve the above problems, the present invention provides an image processing method, device, equipment and medium based on key point detection, which solves the technical problems that when the document format changes or there is non-standard text, recognition errors are prone to occur when using only OCR technology to recognize images, and the post-processing is not robust enough, which may lead to inaccurate or missing information extraction.

[0007] In view of this, a first aspect of the present invention provides an image processing method based on key point detection, the method comprising:

[0008] Acquire image data to be identified;

[0009] Using the image data to be identified as an input of a pre-trained ViTPose model, obtaining first key point data output by the pre-trained ViTPose model, wherein the first key point data includes first coordinate data and a first column label;

[0010] Determine the coordinate range of each column of the image data to be identified according to the first coordinate data of the first key point having the same first column label;

[0011] The characters in each column are recognized according to the coordinate range to obtain a content recognition result including the first column tag.

[0012] Optionally, before acquiring the image data to be identified, the method further includes:

[0013] Acquire training image data, wherein the training image data includes a plurality of second key point data for marking a target column position, wherein the second key point data includes second coordinate data and a second column label;

[0014] Based on a preset loss function and a preset optimizer, the pre-trained ViTPose model is trained using the training image data, and when the preset loss function converges, the trained pre-trained ViTPose model is obtained.

[0015] Optionally, based on a preset loss function and a preset optimizer, the training of the pre-trained ViTPose model using the training image data is specifically as follows:

[0016] The PatchEmbedding layer of the pre-trained ViTPose model divides the image of the training image data into a plurality of slices of target size and performs dimensionality reduction processing to obtain the training image data having a data structure of the number of slices multiplied by the slice dimension;

[0017] The Positional Encoding layer of the pre-trained ViTPose model adds position information to each of the slices by constructing an embedding table at the position of the slices;

[0018] The pre-trained ViTPose model includes a backbone network composed of several series-connected feature extraction layers, each of which extracts image features of each cut image based on a self-attention mechanism, and then performs layer normalization processing through a feed-forward neural network to obtain feature results;

[0019] After the image decoding layer of the pre-trained ViTPose model upsamples the feature results output by the backbone network, an output result in a heat map format is generated through a convolution layer;

[0020] Based on a preset loss function and a preset optimizer, the pre-trained ViTPose model is iteratively trained using a gradient descent method with a preset learning rate and a preset batch size.

[0021] Optionally, the step of using the image data to be identified as an input of a pre-trained ViTPose model to obtain first key point data output by the pre-trained ViTPose model, wherein the first key point data includes first coordinate data and a first column label and specifically includes:

[0022] Inputting the image to be recognized into the trained pre-trained ViTPose model;

[0023] Each of the feature extraction layers of the pre-trained ViTPose model outputs first key point data of the image to be identified, respectively, and the first key point data includes first coordinate data, a first column label and probability data.

[0024] Optionally, before determining the coordinate range of each column of the to-be-identified image data according to the first coordinate data corresponding to the same first column tag, the method further includes:

[0025] Traversing the first key point data layer by layer, and determining the first coordinate data corresponding to the probability data of the maximum value in each of the feature extraction layers;

[0026] The first key point data whose probability data in each feature extraction layer does not meet a preset threshold is eliminated.

[0027] In one embodiment, a feature extraction algorithm (such as SIFT, SURF, ORB, etc.) performs feature extraction on the image data to be identified, identifies the first key point data and its corresponding descriptor, and determines a preset probability threshold for subsequent first key point data screening. Then, each feature extraction layer is traversed, and in each layer, the first key point data with the maximum probability value is found, and the coordinate data of the key point with the maximum probability value in each layer is recorded, and then the first key point data of each layer is screened, and the first key point data whose probability value does not meet the preset threshold is eliminated, and the first key point data that does not meet the conditions is removed.

[0028] Optionally, after removing the first key point data whose probability data in each feature extraction layer does not meet a preset threshold, the method further includes:

[0029] The first key point data output by each of the feature extraction layers is verified layer by layer, wherein:

[0030] If the feature extraction layer is the first layer, retaining the first key point data;

[0031] Otherwise, compare the vertical coordinates of each of the first key point data of the current layer with the minimum vertical coordinate value of the first key point of the previous layer. If the vertical coordinates of each of the first key point data of the current layer are greater than the minimum vertical coordinate value, retain the first key point data of the current layer; otherwise, discard the first key point data of the current layer.

[0032] Optionally, the identifying of the text in each column according to the coordinate range to obtain the content identification result including the first column tag is specifically:

[0033] After performing image segmentation on the image data to be identified according to the coordinate range, a plurality of column segmentation images are obtained according to the first column label corresponding to the coordinate range;

[0034] The OCR recognition module is called to perform text recognition on each of the column cut images to obtain a content recognition result including the first column label.

[0035] In one embodiment, the image segmentation method includes:

[0036] First, clarify the coordinate range of each column, and define the position and size of each column in the image through the coordinates;

[0037] Afterwards, the image is preprocessed to improve the accuracy of subsequent cutting and recognition. The preprocessing process includes adjusting the image contrast, brightness, denoising, rotation correction, etc.

[0038] After that, the image is cut according to the defined coordinate range using image processing software or programming libraries (such as OpenCV, PIL / Pillow, etc.). Each coordinate range will cut out a small image, i.e. a column cut.

[0039] Afterwards, each column cut image is stored as a separate image file for subsequent processing and recognition.

[0040] Afterwards, each cut column image is associated with the corresponding column label to ensure that each image is correctly matched to its label.

[0041] Afterwards, the cut column images are quality checked to ensure that the cutting is accurate and the image content is clear and recognizable.

[0042] In one embodiment, when the OCR recognition module performs text recognition, it includes:

[0043] Identify the boundaries of each segmented region and determine the location and size of each region;

[0044] Apply OCR technology to each segmented area to convert the text in the image into a machine-readable text format. When recognizing, you can choose an appropriate OCR engine, such as Tesseract, Google Vision API, etc.

[0045] Analyze the OCR recognition results and extract the content including the tag of the first column;

[0046] The OCR recognition results are then verified to ensure the accuracy of the recognition.

[0047] A second aspect of the present invention provides an image processing device based on key point detection, the device comprising:

[0048] A first acquisition unit, used for acquiring image data to be recognized;

[0049] A processing unit, configured to use the image data to be identified as an input of a pre-trained ViTPose model to obtain first key point data output by the pre-trained ViTPose model, wherein the first key point data includes first coordinate data and a first column label;

[0050] A coordinate range determining unit, configured to determine a coordinate range of each column of the to-be-recognized image data according to the first coordinate data of the first key point having the same first column label;

[0051] The recognition unit is used to recognize the text in each column according to the coordinate range, and obtain a content recognition result including the first column tag.

[0052] A third aspect of the present invention provides an image processing device based on key point detection, the device comprising a processor and a memory:

[0053] The memory is used to store program code and transmit the program code to the processor;

[0054] The processor is used to execute the steps of the image processing method based on key point detection as described in the first aspect according to the instructions in the program code.

[0055] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0056] It can be seen from the above technical solutions that the embodiments of the present invention have the following advantages:

[0057] In the present invention, an image processing method, device, equipment and medium based on key point detection are provided, which extracts key point data of image data to be identified based on a pre-trained ViTPose model, thereby calibrating the coordinate range of each column in the image data to be identified, and can provide more accurate content recognition and positioning for the OCR recognition module, and finally obtain the content recognition results corresponding to each column. Among them, the pre-trained ViTPose model can capture global context information by relying on the self-attention mechanism, and can more accurately detect the key points in the image data for the long-distance dependencies that may exist between different key points, which solves the technical problems that when the document format changes or there is non-standard text, recognition errors are prone to occur when using only OCR technology to identify pictures, and the post-processing is not robust enough, which may lead to inaccurate or missing information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A method flow chart of an image processing method based on key point detection in an embodiment of the present invention;

[0059] Figure 2 Schematic diagram of the structure of an image processing device based on key point detection in an embodiment of the present invention;

[0060] Figure 3 Schematic diagram of the structure of an image processing device based on key point detection in an embodiment of the present invention; DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The present invention designs an image processing method, device, equipment and medium based on key point detection, which solves the technical problems that when the document format changes or there is non-standard text, recognition errors are prone to occur when using only OCR technology to recognize images, and the robustness of post-processing is insufficient, which may lead to inaccurate or missing information extraction.

[0063] For easier understanding, see Figure 1 , Figure 1 is a method flow chart of an image processing method based on key point detection in an embodiment of the present invention. Figure 1 As shown, specifically:

[0064] S101, obtaining image data to be recognized;

[0065] These image data may come from scanners, cameras or other digitizing devices. It should be noted that the image data to be processed are collected, and these image data are letters of credit or other documents to be identified.

[0066] For example: suppose we have a letter of credit image and need to extract information such as beneficiary name and amount from it.

[0067] S102, using the image data to be recognized as the input of the pre-trained ViTPose model to obtain first key point data output by the pre-trained ViTPose model, where the first key point data includes first coordinate data and a first column label;

[0068] The ViTPose model is a posture estimation model based on the Vision Transformer architecture. It should be noted that the image to be recognized is input into the pre-trained ViTPose model, and the model outputs key point data, including coordinates and column labels. These key point data are the basis for subsequent processing. By analyzing the coordinate distribution of key points with the same column label, the coordinate range of each column is accurately determined.

[0069] Example: The model identifies the location of the "Beneficiary" column on a letter of credit and outputs the coordinates and label.

[0070] The first key point data includes first coordinate data and a first column label, and specifically includes: inputting the image to be identified into the trained pre-trained ViTPose model; each feature extraction layer of the pre-trained ViTPose model respectively outputs the first key point data of the image to be identified, and the first key point data includes first coordinate data, a first column label and probability data.

[0071] The architecture of the pre-trained ViTPose model is based on Vision Transformer, which has good versatility for various visual tasks. Therefore, it can be relatively easily adapted to other key point detection tasks, such as the document column detection in the present invention.

[0072] At the same time, the self-attention mechanism of the VIT model enables the model to capture global context information. This is crucial for understanding complex spatial relationships. In some key point detection tasks, there may be long-distance dependencies between different key points. The self-attention mechanism of the VitPose model helps capture these long-distance dependencies and improve detection accuracy.

[0073] Finally, through transfer learning, the VitPose model may also achieve good performance on smaller datasets, which is helpful for specific key point detection tasks where it is difficult to obtain a large amount of labeled data.

[0074] S103, determining the coordinate range of each column of the image data to be identified according to the first coordinate data of the first key point having the same first column label;

[0075] It should be noted that the coordinate range of each column in the image is determined according to the coordinate data corresponding to the key points having the same column label.

[0076] For example, the model outputs multiple key points of the "beneficiary" column, and these points are used to determine the exact location of the column in the image.

[0077] The first key point data output by the pre-trained ViTPose model obtained in step S102 include the coordinates of the key point (first coordinate data) and the column label (first column label) related thereto.

[0078] Classifying all first key point data according to their column labels, extracting coordinate data of all key points for each column label classification, and calculating the coordinate range for each column label classification, which may include:

[0079] Determine the minimum and maximum abscissas (X coordinates) for each column to determine the horizontal range;

[0080] Determine the minimum and maximum ordinates (Y coordinates) for each column to determine the vertical range;

[0081] Then verify whether the calculated coordinate range is reasonable, ensure that the coordinates are within the boundaries of the image, and can cover all relevant first key points.

[0082] Assuming there is an image with three columns (A, B, C), the ViTPose model outputs the following key point data:

[0083] Key point coordinates of column A: (100,200), (150,200), (100,250), (150,250)

[0084] Key point coordinates of column B: (200,200), (250,200), (200,250), (250,250)

[0085] Key point coordinates of column C: (300,200), (350,200), (300,250), (350,250)

[0086] Therefore, it is necessary to classify the first key point according to the column label. For column A, calculate the coordinate range:

[0087] Minimum horizontal axis: 100

[0088] Maximum horizontal axis: 150

[0089] Minimum vertical coordinate: 200

[0090] Maximum vertical coordinate: 250

[0091] Therefore, the coordinate range of column A is: [100, 150] x [200, 250], and the coordinate ranges of columns B and C are determined in the same way as above.

[0092] S104 , recognizing the text in each column according to the coordinate range, and obtaining a content recognition result including the first column tag.

[0093] It should be noted that when performing text recognition, OCR technology can be used to recognize the column text in the image according to the determined coordinate range.

[0094] For example: OCR technology recognizes the text content in the "Beneficiary" column and converts it into editable text.

[0095] Furthermore, before obtaining the image data to be recognized, the following steps are also included:

[0096] S1001, obtaining training image data, wherein the training image data includes a plurality of second key point data for marking a target column position, wherein the second key point data includes second coordinate data and a second column label;

[0097] S1002. Based on a preset loss function and a preset optimizer, the pre-trained ViTPose model is trained using training image data. When the preset loss function converges, a trained pre-trained ViTPose model is obtained.

[0098] The training process continues until the loss function converges, ensuring that the model fully learns the features in the training data. In this way, the ViTPose model not only inherits the general feature extraction capabilities of the pre-trained model, but also gains a deep understanding of the specific document structure. This combination makes the model perform better when processing various complex documents.

[0099] It should be noted that before actual application, the pre-trained ViTPose model is trained using training image data to adapt to specific image recognition tasks.

[0100] Example: Using a training dataset containing images and annotations of letters of credit, train a model to recognize specific columns on a letter of credit.

[0101] The image to be recognized is input into the trained model, and the model outputs the coordinates, labels, and probabilities of each key point.

[0102] For example: The model processes a letter of credit image and outputs the key points of each column, including coordinates, labels, and recognition probabilities.

[0103] Furthermore, step S1002 is specifically as follows:

[0104] S10021. The Patch Embedding layer of the pre-trained ViTPose model divides the image of the training image data into a number of slices of target size and performs dimensionality reduction processing to obtain training image data whose data structure is the number of slices multiplied by the slice dimension;

[0105] S10022. The Positional Encoding layer of the pre-trained ViTPose model builds an embedding table at the position of the slice to add position information to each slice;

[0106] S10023. The pre-trained ViTPose model includes a backbone network consisting of several serially connected feature extraction layers. After each feature extraction layer extracts the image features of each cut image based on the self-attention mechanism, it performs layer normalization processing through a feed-forward neural network to obtain the feature results.

[0107] S10024, after the image decoding layer of the pre-trained ViTPose model upsamples the feature results output by the backbone network, it generates the output results in the heat map format through the convolution layer;

[0108] S10025. Based on the preset loss function and the preset optimizer, the pre-trained ViTPose model is iteratively trained using the gradient descent method with a preset learning rate and a preset batch size.

[0109] It should be noted that the pre-trained ViTPose model mainly consists of the following parts:

[0110] Patch Embedding layer: This layer connects a linear connection layer to a fixed-size image after segmentation and dimensionality reduction operations, and converts the image into an n×d data structure, where n represents the number of slices and d is the dimension of a single slice after transformation.

[0111] For example, if the input image size is 224x224, and the image is divided into fixed-size blocks of 16x16, then 196 blocks will be generated for each image, that is, the input sequence length is 196, the dimension of each block is 16×16×3=768, and the dimension of the linear projection layer is 768×N (N=768). Therefore, the dimension of the input after passing through the linear projection layer is still 196×768, that is, there are a total of 196 tokens, and the dimension of each token is 768.

[0112] Positional Encoding layer: This layer mainly converts the position of the corresponding block of the PatchEmbedding layer by constructing an n×d learnable embedding table. The value of the table is added to the result of the PatchEmbedding layer and output to the next layer.

[0113] Feature extraction layer: includes attention mechanism layer, feedforward propagation layer and short connection layer.

[0114] The attention mechanism layer contains three kernel functions q, k, and v. v, k, q perform matrix operations with the image features of the Positional Encoding layer and output v out ,k out ,q out .

[0115] Self-attention layer: for v in the feature extraction layer out With k out The matrix is ​​transposed and normalized row by row through softmax to get the matrix. out The matrix multiplication obtains the result of the self-attention layer. Then, after the layer is normalized through the forward neural network, it is accumulated with the short connection layer and the attention input vector to obtain the final result. At this point, the single-layer construction is completed, and the single layer is expanded in series to form a feature extraction layer.

[0116] Backbone network: connect the feature extraction layers in series to form a backbone network.

[0117] Image decoding layer: It is mainly composed of deconvolution layer and 1x1 convolution layer. It upsamples the output of the backbone network. The number of 1x1 convolution kernels corresponds to the total number of columns in the letter of credit. The output is in heatmap format.

[0118] The sorted process is as follows:

[0119] The input image passes through the PatchEmbedding layer, which divides the image into small patches and reduces the dimensionality;

[0120] The Positional Encoding layer adds position information to each small block;

[0121] The feature extraction layer extracts image features through the attention mechanism;

[0122] The self-attention layer further processes the features and obtains the final features through a feed-forward neural network and layer normalization;

[0123] Connect the feature extraction layers in series to form a backbone network;

[0124] The image decoding layer upsamples the output of the backbone network and generates a heatmap through a 1x1 convolutional layer to indicate the location of the key points.

[0125] In the training process of machine learning models, loss function and optimizer are two key components, which together determine the efficiency of model training and the final performance.

[0126] In the embodiment of the present invention, the preset loss function adopts the SmoothL1 loss function, which is a commonly used loss function, particularly suitable for regression problems. It is expressed as L2 loss (square loss) when the error is small, and as L1 loss (absolute value loss) when the error is large. This feature makes it more robust than the simple L2 loss when dealing with outliers.

[0127] In the embodiment of the present invention, the preset optimizer adopts the Adam optimizer. Adam (Adaptive Moment Estimation) is a very popular optimizer, which combines the ideas of momentum and RMSprop (Root Mean Square Propagation) and can automatically adjust the learning rate of each parameter.

[0128] The training process of the pre-trained ViTPose model in the embodiment of the present invention is as follows:

[0129] initialization:

[0130] The model weights are initialized randomly.

[0131] Set the initial learning rate and batch size.

[0132] Forward propagation:

[0133] For each batch of data, the model performs forward calculations to obtain predicted values.

[0134] Calculate the loss:

[0135] The Smooth L1 loss function is used to calculate the error between the predicted value and the true value.

[0136] Backward Propagation:

[0137] The gradient is calculated according to the loss function and the weights of the model are updated through back-propagation.

[0138] Optimizer updates:

[0139] The Adam optimizer updates the model weights based on the calculated gradients and the adaptive learning rate.

[0140] Iteration:

[0141] Repeat the above steps until the performance of the model on the validation set no longer improves, that is, the model converges.

[0142] Save the model:

[0143] After training is completed, save the model parameters for subsequent deployment and use.

[0144] Further, the image data to be identified is used as the input of the pre-trained ViTPose model to obtain the first key point data output by the pre-trained ViTPose model, and the first key point data includes the first coordinate data and the first column label specifically including:

[0145] Input the image to be recognized into the pre-trained ViTPose model;

[0146] Each feature extraction layer of the pre-trained ViTPose model outputs the first key point data of the image to be identified, and the first key point data includes first coordinate data, first column label and probability data.

[0147] Furthermore, before step S103, the following steps are also included:

[0148] S1021, traversing the first key point data layer by layer, and determining the first coordinate data corresponding to the probability data of the maximum value in each feature extraction layer;

[0149] S1022: Eliminate the first key point data in each feature extraction layer whose probability data does not meet a preset threshold.

[0150] It should be noted that in order to ensure that the identified key points have high confidence and improve accuracy, for each layer output, the point with the highest probability value, that is, the maximum value position (x, y) is found.

[0151] Only keypoints with probability values ​​greater than 0.6 are retained, and these points are considered to be potential valid keypoints.

[0152] For each key point, further verification is performed to confirm its authenticity.

[0153] In a specific embodiment, suppose there is a document image containing multiple columns such as title, text, footer, etc. The pre-trained ViTPose model outputs multiple layers of feature extraction results, each layer containing multiple key point data. When performing image processing, first traverse each layer to find the key point with the highest probability value. For example, in the first layer, a key point with a probability of 0.95 may be found, corresponding to the upper left corner of the title bar; then a preset threshold is set, such as 0.7, so all key points with probability values ​​below 0.7 will be eliminated, which may include some misidentified background noise or blurred column boundaries.

[0154] Furthermore, after step S1022, the following steps are further included:

[0155] S1023, verifying the first key point data output by each feature extraction layer layer by layer, wherein:

[0156] If the feature extraction layer is the first layer, the first key point data is retained;

[0157] Otherwise, compare the vertical coordinates of each first key point data of the current layer with the minimum vertical coordinate value of the first key point of the previous layer. If the vertical coordinates of each first key point data of the current layer are greater than the minimum vertical coordinate value, retain the first key point data of the current layer; otherwise, discard the first key point data of the current layer.

[0158] It should be noted that the consistency and accuracy of key points are ensured through inter-layer comparison.

[0159] If the first layer is currently being processed, directly save the key point information and continue processing the next layer.

[0160] For non-first layers, find the keypoint coordinates y of the nearest layer from the upper layer min .

[0161] Compare the keypoint coordinates y predicted by the current layer with y min .

[0162] If y is greater than y min , then keep the key point; if less than, delete it.

[0163] Repeat this process until all layers have been processed.

[0164] It is understandable that the pre-trained ViTPose model consists of multiple layers. These layers can be convolutional layers, fully connected layers, recurrent layers, or other types of neural network layers. In the key point detection model, each layer is responsible for processing different features of the input image and outputting information related to these features.

[0165] In the context of keypoint detection, each layer may output a heatmap that represents the probability distribution of the location of each keypoint in the image. Each layer of the model may have different predictions for the location of the keypoints, so these predictions need to be integrated and verified to determine the final keypoint location.

[0166] Furthermore, step S104 is specifically as follows:

[0167] S1041, after performing image segmentation on the image data to be recognized according to the coordinate range, obtain a plurality of column segmentation images according to the first column label corresponding to the coordinate range;

[0168] S1042: Calling an OCR recognition module to perform text recognition on each column cut image to obtain a content recognition result including a first column label.

[0169] The OCR technology is used to perform text recognition on the cut images of each column, which fully utilizes the advantages of existing mature technologies. By performing OCR processing on each column separately, the present invention can use the most suitable recognition algorithm for different types of content (such as text, numbers, tables, etc.), thereby improving the overall recognition accuracy.

[0170] It should be noted that the specific area of ​​each column is determined according to the key point coordinates. Specifically: using the key point information, the area of ​​each column is divided in the order of coordinates from small to large.

[0171] For example, by comparing the coordinate difference of the key points 45A and 46A, the content interval of the column 45A is determined.

[0172] Perform OCR recognition on each column area and output the recognition results in a structured manner.

[0173] The original image is cut according to the areas of each divided column, and each cut corresponds to a column.

[0174] Call OCR technology to perform text recognition on each cut image.

[0175] Fill the recognized text into the corresponding column in the order of the columns.

[0176] After all fields are identified and filled, the structured data is output.

[0177] The advantages of the image processing method based on key point detection of the present invention are mainly reflected in the following aspects:

[0178] Improved recognition accuracy: By comprehensively considering the output of multiple feature extraction layers and the probability of each key point, the system can more accurately locate individual columns and structural elements in the document.

[0179] Enhanced system robustness: Even when dealing with documents of poor quality or complex structures, the system can screen and verify key points through probabilistic data, thereby maintaining a high recognition accuracy rate.

[0180] Provides richer information: Probabilistic data provides additional reference for subsequent processing, allowing the system to adopt more flexible and intelligent strategies to handle different situations.

[0181] Optimized processing flow: By obtaining high-quality key point data at an early stage, the subsequent column positioning and text recognition process can be more efficient and accurate.

[0182] In general, the present invention effectively improves the accuracy and reliability of document image processing by making full use of the characteristics of the pre-trained visual pose estimation model and introducing the key information of probability data. This method can not only better cope with complex document structures, but also provides a solid foundation for subsequent text recognition and data extraction. Through this innovation, the present invention has made significant progress in the field of document image processing and made an important contribution to improving the automation level and efficiency of document processing.

[0183] See also Figure 2 , Figure 2 FIG. 1 is a schematic diagram of the structure of an image processing device based on key point detection in an embodiment of the present invention. Figure 2 As shown, specifically:

[0184] A first acquisition unit 201 is used to acquire image data to be recognized;

[0185] The processing unit 202 is used to use the image data to be identified as the input of the pre-trained ViTPose model to obtain the first key point data output by the pre-trained ViTPose model, where the first key point data includes the first coordinate data and the first column label;

[0186] A coordinate range determining unit 203, configured to determine the coordinate range of each column of the image data to be identified according to the first coordinate data of the first key point having the same first column label;

[0187] The recognition unit 204 is used to recognize the text in each column according to the coordinate range, and obtain the content recognition result including the first column tag.

[0188] Furthermore, it also includes:

[0189] A second acquisition unit is used to acquire training image data, wherein the training image data includes a plurality of second key point data for marking the position of a target column, and the second key point data includes second coordinate data and a second column label;

[0190] The training unit is used to train the pre-trained ViTPose model using training image data based on a preset loss function and a preset optimizer, and obtain the trained pre-trained ViTPose model when the preset loss function converges.

[0191] Furthermore, the training unit is specifically used for:

[0192] The PatchEmbedding layer of the pre-trained ViTPose model divides the image of the training image data into several slices of target size and performs dimensionality reduction processing to obtain the training image data with a data structure of the number of slices multiplied by the slice dimension;

[0193] The Positional Encoding layer of the pre-trained ViTPose model adds position information to each slice by constructing an embedding table at the slice position;

[0194] The pre-trained ViTPose model contains a backbone network consisting of several series-connected feature extraction layers. Each feature extraction layer extracts the image features of each slice based on the self-attention mechanism, and then performs layer normalization processing through the forward neural network to obtain the feature results.

[0195] The image decoding layer of the pre-trained ViTPose model upsamples the feature results output by the backbone network and generates the output results in heat map format through the convolution layer;

[0196] Based on the preset loss function and the preset optimizer, the pre-trained ViTPose model is iteratively trained using the gradient descent method with the preset learning rate and the preset batch size.

[0197] Furthermore, the processing unit 202 is specifically configured to:

[0198] Input the image to be recognized into the pre-trained ViTPose model;

[0199] Each feature extraction layer of the pre-trained ViTPose model outputs the first key point data of the image to be identified, and the first key point data includes first coordinate data, first column label and probability data.

[0200] Furthermore, a verification unit is included, which is used to:

[0201] Traversing the first key point data layer by layer, determining the first coordinate data corresponding to the probability data of the maximum value in each feature extraction layer;

[0202] Eliminate the first key point data in each feature extraction layer whose probability data does not meet the preset threshold.

[0203] Furthermore, the verification unit is also used to:

[0204] Verify the first key point data output by each feature extraction layer layer by layer, where:

[0205] If the feature extraction layer is the first layer, the first key point data is retained;

[0206] Otherwise, compare the vertical coordinates of each first key point data of the current layer with the minimum vertical coordinate value of the first key point of the previous layer. If the vertical coordinates of each first key point data of the current layer are greater than the minimum vertical coordinate value, retain the first key point data of the current layer; otherwise, discard the first key point data of the current layer.

[0207] Furthermore, the identification unit is specifically used for:

[0208] After the image data to be recognized is segmented according to the coordinate range, a number of segmented images are obtained according to the first column label corresponding to the coordinate range;

[0209] The OCR recognition module is called to perform text recognition on each column cut image to obtain a content recognition result including the label of the first column.

[0210] Another embodiment of the present invention provides an image processing device based on key point detection, such as Figure 3 As shown, the device 10 includes:

[0211] One or more processors 110 and memory 120, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0212] The processor 110 is used to complete various control logics of the device 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0213] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for constructing a multilingual phoneme representation model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for constructing a multilingual phoneme representation model in the above method embodiment.

[0214] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0215] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0216] Acquire image data to be identified;

[0217] Using the image data to be identified as the input of the pre-trained ViTPose model, obtaining the first key point data output by the pre-trained ViTPose model, where the first key point data includes the first coordinate data and the first column label;

[0218] Determine the coordinate range of each column of the image data to be identified according to the first coordinate data of the first key point having the same first column label;

[0219] Based on the OCR recognition module, the text in each column is recognized according to the coordinate range, and the content recognition result including the label of the first column is obtained.

[0220] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, any implementation of the image processing method based on key point detection described in the above embodiments is implemented.

[0221] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0222] In an embodiment of the present invention, an image processing method, apparatus, device and medium based on key point detection are provided, which extracts key point data of image data to be identified based on a pre-trained ViTPose model, thereby calibrating the coordinate range of each column in the image data to be identified, and can provide more accurate content recognition and positioning for the OCR recognition module, and finally obtain the content recognition results corresponding to each column. Among them, the pre-trained ViTPose model relies on the self-attention mechanism to capture global context information, and can more accurately detect key points in image data for possible long-distance dependencies between different key points, which solves the technical problems that when the document format changes or there is non-standard text, recognition errors are prone to occur when using only OCR technology to identify pictures, and the post-processing is not robust enough, which may lead to inaccurate or missing information extraction.

[0223] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0224] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0225] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0226] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0227] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0228] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0229] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0230] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0231] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. An image processing method based on key point detection, characterized in that: include: Obtaining image data to be recognized; Using the image data to be identified as an input of a pre-trained ViTPose model, obtaining first key point data output by the pre-trained ViTPose model, wherein the first key point data includes first coordinate data and a first column label; Determine the coordinate range of each column of the image data to be identified according to the first coordinate data of the first key point having the same first column label; The characters in each column are recognized according to the coordinate range to obtain a content recognition result including the first column tag.

2. The image processing method based on key point detection according to claim 1, characterized in that: The step of obtaining the image data to be identified also includes: Acquire training image data, wherein the training image data includes a plurality of second key point data for marking a target column position, wherein the second key point data includes second coordinate data and a second column label; Based on a preset loss function and a preset optimizer, the pre-trained ViTPose model is trained using the training image data, and when the preset loss function converges, the trained pre-trained ViTPose model is obtained.

3. The image processing method based on key point detection according to claim 2, characterized in that: The method of training the pre-trained ViTPose model based on the preset loss function and the preset optimizer using the training image data is specifically as follows: The Patch Embedding layer of the pre-trained ViTPose model divides the image of the training image data into a plurality of slices of target size and performs dimensionality reduction processing to obtain the training image data whose data structure is the number of slices multiplied by the slice dimension; The Positional Encoding layer of the pre-trained ViTPose model adds position information to each of the slices by constructing an embedding table at the position of the slices; The pre-trained ViTPose model includes a backbone network composed of several series-connected feature extraction layers, each of which extracts image features of each cut image based on a self-attention mechanism, and then performs layer normalization processing through a feed-forward neural network to obtain feature results; After the image decoding layer of the pre-trained ViTPose model upsamples the feature results output by the backbone network, an output result in a heat map format is generated through a convolution layer; Based on a preset loss function and a preset optimizer, the pre-trained ViTPose model is iteratively trained using a gradient descent method with a preset learning rate and a preset batch size.

4. The image processing method based on key point detection according to claim 1, characterized in that: The step of using the image data to be identified as the input of the pre-trained ViTPose model to obtain the first key point data output by the pre-trained ViTPose model, wherein the first key point data includes the first coordinate data and the first column label, and specifically includes: Inputting the image to be recognized into the trained pre-trained ViTPose model; Each of the feature extraction layers of the pre-trained ViTPose model outputs first key point data of the image to be identified, respectively, and the first key point data includes first coordinate data, a first column label and probability data.

5. The image processing method based on key point detection according to claim 4, characterized in that: Before determining the coordinate range of each column of the to-be-identified image data according to the first coordinate data corresponding to the same first column tag, the method further includes: Traversing the first key point data layer by layer, and determining the first coordinate data corresponding to the probability data of the maximum value in each of the feature extraction layers; The first key point data whose probability data in each feature extraction layer does not meet a preset threshold is eliminated.

6. The image processing method based on key point detection according to claim 5, characterized in that: After removing the first key point data whose probability data in each feature extraction layer does not meet a preset threshold, the method further includes: The first key point data output by each of the feature extraction layers is verified layer by layer, wherein: If the feature extraction layer is the first layer, retaining the first key point data; Otherwise, compare the vertical coordinates of each of the first key point data of the current layer with the minimum vertical coordinate value of the first key point of the previous layer. If the vertical coordinates of each of the first key point data of the current layer are greater than the minimum vertical coordinate value, retain the first key point data of the current layer; otherwise, discard the first key point data of the current layer.

7. The image processing method based on key point detection according to claim 1, characterized in that: The text in each column is recognized according to the coordinate range, and the content recognition result including the first column tag is obtained as follows: After performing image segmentation on the image data to be identified according to the coordinate range, a plurality of column segmentation images are obtained according to the first column label corresponding to the coordinate range; The OCR recognition module is used to perform text recognition on each of the column cut images to obtain a content recognition result including the first column label.

8. An image processing device based on key point detection, characterized in that: include: A first acquisition unit, used for acquiring image data to be recognized; A processing unit, configured to use the image data to be identified as an input of a pre-trained ViTPose model to obtain first key point data output by the pre-trained ViTPose model, wherein the first key point data includes first coordinate data and a first column label; A coordinate range determining unit, configured to determine a coordinate range of each column of the to-be-recognized image data according to the first coordinate data of the first key point having the same first column label; The recognition unit is used to recognize the text in each column according to the coordinate range, and obtain a content recognition result including the first column tag.

9. An image processing device based on key point detection, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the image processing method based on key point detection according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the image processing method based on key point detection according to any one of claims 1 to 7.

Images