Image data information extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image data information extraction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring to-be-processed initial image data; inputting the initial image data into a trained SAM network model, and outputting first pre-segmented image data corresponding to a text information area in the initial image data; and inputting the first pre-segmented image data into a trained CRNN network model, and outputting a first text detection result corresponding to the initial image data. According to the embodiment of the invention, an SAM + CRNN algorithm is adopted, pre-segmentation is carried out through arbitrary segmentation performance of the SAM, the text information part in the business opportunity picture is positioned more accurately, and then the CRNN is used for character recognition, so that the extraction of the text information is more complete and more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and in particular to an information extraction method and device for image data, an electronic device and a storage medium. BACKGROUND

[0002] In the process of uploading business opportunity pictures on business opportunity information, the pictures taken by front-line sales personnel on mobile phones will present any angle, and non-standard shooting will lead to the fact that traditional OCR (Optical Character Recognition) technology cannot accurately identify the text information in the pictures. SUMMARY

[0003] In view of the above problems, the present application provides an information extraction method and device for image data, an electronic device and a storage medium to overcome the above problems or at least partially solve the above problems, comprising:

[0004] An information extraction method for image data, the method comprising:

[0005] obtaining initial image data to be processed;

[0006] inputting the initial image data into a trained SAM network model to output first pre-segmentation image data corresponding to a text information region of the initial image data;

[0007] inputting the first pre-segmentation image data into a trained CRNN network model to output a first text detection result corresponding to the initial image data.

[0008] Optionally, the step of inputting the initial image data into the trained SAM network model to output the first pre-segmentation image data corresponding to the text information region of the initial image data comprises:

[0009] inputting the initial image data into the trained SAM network model to determine an image feature vector corresponding to the initial image data;

[0010] obtaining prompt information corresponding to the SAM network model and determining a prompt vector corresponding to the prompt information;

[0011] performing scene graph pre-segmentation on the initial image data based on the prompt vector and the image feature vector to output the first pre-segmentation image data corresponding to the text information region of the initial image data.

[0012] Optionally, the step of performing scene graph pre-segmentation on the initial image data based on the prompt vector and the image feature vector to output the first pre-segmentation image data corresponding to the text information region of the initial image data comprises:

[0013] perform scene graph pre-segmentation on the initial image data based on the prompt vector and the image feature vector, to generate a segmentation mask;

[0014] optimize the segmentation mask according to a preset mask optimization manner in the SAM network model, and output first pre-segmentation image data corresponding to a text information region of the initial image data.

[0015] Optionally, the inputting the pre-segmentation image data into the trained CRNN network model and outputting a text detection result corresponding to the initial image data comprises:

[0016] inputting the pre-segmentation image data into the trained CRNN network model, and pre-processing the pre-segmentation image data by using the CRNN network model to obtain first image data;

[0017] extracting features of the first image data by using a CNN structure in the CRNN network model to obtain deep feature data;

[0018] generating a first text detection result corresponding to the initial image data based on the deep feature data by using an RNN structure in the CRNN network model.

[0019] Optionally, the inputting the pre-segmentation image data into the trained CRNN network model and pre-processing the pre-segmentation image data by using the CRNN network model to obtain first image data comprises:

[0020] inputting the pre-segmentation image data into the trained CRNN network model, and converting the pre-segmentation image data into gray image data;

[0021] generating binary image data based on the gray image data;

[0022] pre-processing the binary image data according to a preset format specification to obtain first image data.

[0023] Optionally, the pre-processing the binary image data according to a preset format specification to obtain first image data comprises:

[0024] performing denoising processing on the binary image data to obtain first image data;

[0025] and / or, detecting a direction of a text line of the binary image data and correcting an inclination to obtain first image data;

[0026] and / or, adjusting the binary image data to a preset input size and then performing normalization processing to obtain first image data.

[0027] Optionally, the method further comprises:

[0028] obtain a sample data set for training a network model, the sample data set comprising sample picture data and original text labels corresponding to the sample picture data;

[0029] construct an initial SAM network structure and an initial CRNN network structure;

[0030] input the sample picture data into the initial SAM network structure to obtain second pre-segmented image data

[0031] input the second pre-segmented image data and the original text labels into the initial CRNN network structure to output second text detection results;

[0032] determine a loss function based on the second text detection results and the original text labels;

[0033] iteratively train the initial SAM network structure and the initial CRNN network structure according to the loss function.

[0034] An image detection device, the device comprising:

[0035] an initial image data obtaining module configured to obtain initial image data to be processed;

[0036] a first pre-segmented image data generating module configured to input the initial image data into a trained SAM network model to output first pre-segmented image data corresponding to a text information region of the initial image data;

[0037] a first text detection structure generating module configured to input the first pre-segmented image data into a trained CRNN network model to output first text detection results corresponding to the initial image data.

[0038] An electronic device comprising a processor, a memory, and a computer program stored on the memory and capable of running on the processor, the computer program being executed by the processor to implement the image data information extraction method as described above.

[0039] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the image data information extraction method as described above.

[0040] Embodiments of the present application have the following advantages:

[0041] The embodiment of the present application obtains initial image data to be processed; inputs the initial image data into a trained SAM network model, outputs first pre-segmentation image data corresponding to a Chinese text information region of the initial image data, and inputs the first pre-segmentation image data into a trained CRNN network model, and outputs a first text detection result corresponding to the initial image data, so as to realize the SAM+CRNN algorithm, pre-segmentation by the arbitrary segmentation performance of SAM, more accurate positioning of the Chinese text information part of the business opportunity picture, and then text recognition by the CRNN network, so that the extraction of the text information is more complete and more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the description of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a step flow chart of an image data information extraction method provided by an embodiment of the present application;

[0044] Figure 2 is a step flow chart of another image data information extraction method provided by an embodiment of the present application;

[0045] Figure 3 is a step flow chart of another image data information extraction method provided by an embodiment of the present application;

[0046] Figure 4 is a step flow chart of a business opportunity picture processing method provided by an embodiment of the present application;

[0047] Figure 5 is a structural schematic diagram of an image data information extraction device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following will further describe the present application in combination with the drawings and specific embodiments. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0049] Referring to Figure 1 , a step flow chart of an image data information extraction method provided by an embodiment of the present application is shown, which can specifically include the following steps:

[0050] Step S101, obtain initial image data to be processed;

[0051] In the embodiment of the present application, initial image data to be processed can be obtained, which can be used for text content extraction. In actual application, the initial image data can be a business opportunity picture, for example, the initial image data uploaded by iSee in the pen-and-paper customer business opportunity entry function.

[0052] Step S102, input the initial image data into the trained SAM network model, and output the first pre-segmentation image data corresponding to the text information region of the initial image data;

[0053] In the embodiment of the present application, a SAM network model can be pre-trained, which is used to realize image pre-segmentation. The input data of the SAM network model is the image data to be processed, and the output data is the first pre-segmentation image data after segmentation.

[0054] Among them, SAM (Segment Anything Model) is an advanced image segmentation model, which is used to identify and segment any object in an image, and can process multiple image segmentation tasks through prompt learning. SAM uses an efficient image encoder and a lightweight mask decoder to output high-quality segmentation masks.

[0055] The technical background of SAM (Segment Anything Model) is built on several key aspects:

[0056] (1) Large-scale dataset:

[0057] To train this model, a large-scale dataset was constructed, containing over 11 million copyrighted images and over 10 billion segmentation masks, which is the largest segmentation dataset to date.

[0058] (2) Prompt segmentation tasks:

[0059] The SAM model is designed to handle so-called "prompt segmentation" tasks. In this task, the model needs to generate the corresponding segmentation mask according to the simple prompt provided by the user (such as a point, a box or a piece of text).

[0060] (3) Zero-shot transfer learning:

[0061] The SAM model is designed to achieve zero-shot transfer learning through prompt engineering technology, which means it can directly handle new image distributions and tasks without training for specific tasks.

[0062] (4) Model architecture:

[0063] The SAM model includes an image encoder, a cue encoder, and a lightweight mask decoder. This design enables the model to flexibly handle different types of cues and quickly generate segmentation masks.

[0064] The advantages of the embodiments of the present application over the prior art are that background interference is effectively reduced, i.e., the pre-segmentation of SAM can more accurately cover the identified text target, reduce the interference of picture background and irrelevant information, and improve the recognition accuracy.

[0065] Step S103, input the first pre-segmentation image data into the trained CRNN network model, and output the first text detection result corresponding to the initial image data.

[0066] After obtaining the pre-segmentation image data, the first pre-segmentation image data can be input into the trained CRNN network model to further implement image text recognition, so as to finally output the first text detection result corresponding to the initial image data.

[0067] Among them, CRNN (Convolutional Recurrent Neural Network) is a deep learning model for sequence recognition, which combines the image feature extraction ability of convolutional neural network (CNN) and the sequence processing ability of recurrent neural network (RNN), and is suitable for scene text recognition and other tasks. CRNN model combines the advantages of convolutional neural network (CNN) and recurrent neural network (RNN) to process variable-length sequence data. Its technical background is established in several key aspects:

[0068] (1) Development of deep learning:

[0069] In the field of computer vision, image-based sequence recognition has always been an important research topic. This includes the recognition of sequence-like visual objects such as scene text, handwriting, and music scores. These objects usually appear in sequence form rather than isolated single entities. With the success of deep learning, especially deep convolutional neural networks (DCNN) in various visual tasks, researchers began to explore the application of deep learning to sequence recognition problems. However, DCNN models usually require fixed-size inputs and outputs, which are not suitable for directly processing variable-length sequences.

[0070] (2) Introduction of CRNN model:

[0071] The CRNN model proposes a new neural network architecture by integrating CNN and RNN, which can be trained end-to-end without complex preprocessing steps. The CRNN model can directly extract feature sequences from images and use RNN to process these sequences to generate sequence labels.

[0072] (3) Features of CRNN:

[0073] Process sequences of arbitrary length: CRNN can process sequences of arbitrary length without character segmentation or horizontal scale normalization.

[0074] Not limited by predefined dictionary: CRNN can achieve excellent performance without a dictionary, suitable for out-of-vocabulary text recognition tasks.

[0075] Generate smaller models: CRNN model has fewer parameters, compared to traditional DCNN models, it is more compact and more suitable for practical application scenarios.

[0076] In the embodiments of the present application, the CRNN model can be used to process the first pre-segmented data of various initial images taken by the user for unified text detection, which has a wide range of applications and high accuracy in extracting text information.

[0077] In an embodiment of the present application, the training process of the SAM network model and the CRNN network model can further include the following sub-steps:

[0078] Sub-step S11, obtaining a sample data set for training the network model, the sample data set including sample picture data and original text labels corresponding to the sample picture data;

[0079] In the embodiments of the present application, a large number of opportunity upload photos can be obtained from the historical files of a preset file server (for example, iSee file server) as a data set, and the pictures in the data set can contain various angles, environmental backgrounds, and text size types. In actual applications, batch image processing can be used to convert them into a unified size of trainable image data set.

[0080] After obtaining the trainable image data set, a marking tool can be used to manually annotate the key text information such as numbers in the data set to ensure that the mask box covers all information, transcribe the text content in the image into a text string, and use it as model training data (i.e. sample data set). The sample data set can include sample picture data and original text labels of the sample image data, and the original text labels correspond to the text contained in the sample picture data.

[0081] In practical applications, in order to improve the model recognition accuracy, data enhancement operations such as rotation, scaling, cropping and adding random noise can also be performed on the image. After data processing, the image does not need to be manually labeled again, and the new position data is calculated according to the affine transformation matrix calculated according to the affine transformation principle of the image.

[0082] In an embodiment of the present application, a picture data set labeled with opportunity information covering real text masks is collected for training to ensure that the picture covers a variety of background environments and lighting conditions. The data set is divided into a training set and a test set, with an occupancy ratio of 80% and 20% respectively. In addition, data enhancement strategies such as random rotation, scaling, cropping and adding random noise are used to improve the robustness and accuracy of the model for extracting opportunity information under different scenes and conditions.

[0083] Substep S12, constructing an initial SAM network structure and an initial CRNN network structure;

[0084] In practical applications, the information extraction system can pre-construct an initial SAM network structure and an initial CRNN network structure. The information extraction system can also include a preprocessing module, which can be used to process input images and can include scaling processing, normalization processing, etc. to adjust the image to a size suitable for model training.

[0085] When constructing the initial SAM network structure, a suitable pre-trained model such as VIT_B, VIT_H, VIT_L, etc. can be selected as the image encoder of SAM to extract image features and further realize image data segmentation. A CRNN module is added after the SAM network to process the text region segmented by SAM and output sequence information. Specifically, a suitable pre-trained model can be selected as the initial CRNN network structure for training to realize image data extraction.

[0086] Substep S13, inputting sample picture data into the initial SAM network structure to obtain second pre-segmented image data;

[0087] After constructing the initial SAM network structure and the initial CRNN network structure, the sample picture data can be output to the initial SAM network structure for image segmentation to obtain second pre-segmented image data.

[0088] Substep S14, inputting the second pre-segmented image data and the original text label into the initial CRNN network structure to output a second text detection result;

[0089] After obtaining the second pre-segmentation image data, the obtained second pre-segmentation image data and the original text label can be input into the initial CRNN network structure to perform text information extraction, and a second text detection result is output, which is the text information extracted by the initial SAM network structure and the initial CRNN network structure for the sample picture.

[0090] In substep S15, a loss function is determined based on the second text detection result and the original text label.

[0091] In actual applications, the initially constructed SAM network structure and CRNN network structure are general network structures, and in order to improve accuracy, the SAM network structure and the CRNN network structure can be iteratively trained based on a sample data set, thereby improving the accuracy of text extraction.

[0092] Specifically, a loss function can be constructed, and then the second text detection result is prediction data, and the original text label is actual extracted information, and the optimization goal is to make the second text detection result obtained by the optimized SAM network structure and CRNN network structure approach the original text label, and the optimization manner is to construct a loss function based on the second text detection result and the original text label, and then realize network structure optimization.

[0093] In substep S16, the initial SAM network structure and the initial CRNN network structure are iterated according to the loss function.

[0094] In an embodiment of the present application, the loss function can be used to define a loss function to optimize network parameters, and the loss function can use a segmentation loss and a sequence loss, the segmentation loss uses Dice Loss:

[0095]

[0096] wherein X is a prediction mask, Y is a ground truth, and the calculation value range is between [0, 1]. The sequence loss uses CTCloss:

[0097]

[0098] wherein S is a training sample subset, x is the output of the original data in the training sample set S after the network Nw, and z is the label corresponding to x. The greater the overall likelihood function, the better the result, and the smaller the loss function, the better the result, so a negative sign is added.

[0099] The training process in the embodiment of the application can be: the model inputs the processed picture of unified specification, first performs pre-segmentation through the basic SAM network architecture, and outputs the business opportunity picture text information region. Then the pre-segmented image and the original text label are transmitted to the CRNN for recognition training, and the sequence label output by the CRNN is generated into the final text line detection result by using the CTC decoder. The Dice Loss and the CTC Loss are calculated respectively, and the overall network loss is the sum of the two losses. The gradient descent algorithm such as Adam or SGD is applied to train the network.

[0100] Model evaluation: evaluate the performance of the model on the validation set, and use indicators such as mAP (mean Average Precision) to measure the accuracy of the model.

[0101] Model inference: use the trained model to infer the test image that has not been seen before, and get the detection result.

[0102] Model optimization: evaluate the current effect of the model, compare the model inference result with the standard answer manually labeled, calculate the accuracy, recall rate, F1 score and mAP of the model, and find out the optimization direction. Adjust the network structure, learning rate, batch size, and labeled data set to improve the recognition effect.

[0103] In the embodiment of the application, the initial image data to be processed is obtained; the initial image data is input into the trained SAM network model, and the first pre-segmented image data corresponding to the text information region of the initial image data is output, and the first pre-segmented image data is input into the trained CRNN network model, and the first text detection result corresponding to the initial image data is output, so as to realize the SAM+CRNN algorithm, pre-segmentation through the arbitrary segmentation performance of SAM, more accurate positioning of the business opportunity picture text information part, and then using the CRNN network for character recognition, so that the extraction of text information is more complete and accurate.

[0104] Referring to Figure 2 , a step flowchart of another information extraction method of image data provided by an embodiment of the application is shown, which can specifically include the following steps:

[0105] Step S201, obtaining initial image data to be processed;

[0106] Step S202, inputting the initial image data into the trained SAM network model to determine the image feature vector corresponding to the initial image data;

[0107] Step S203, obtaining prompt information corresponding to the SAM network model, and determining a prompt vector corresponding to the prompt information;

[0108] In the embodiment of the present application, the user can input prompt information in the SAM network model, and the prompt information can be used to realize segmentation of image data.

[0109] For example, when a business opportunity picture is segmented by using SAM, the method of Prompt can be used to locate the effective information of the business opportunity picture. Specifically, three kinds of prompt enhancements can be provided for scene graph pre-segmentation: 1) random dotting, randomly marking positive sample points and negative sample points according to the foreground and background of key information in the business opportunity picture; 2) proportion dotting, marking sample points by adjusting the number of edge dotting and the number of center dotting to 7:3 according to the key information mask in the business opportunity picture; and 3) dot and frame prompting, using double prompt enhancement of frame + dot, first roughly positioning the effective information position in the business opportunity picture by using a rectangular frame, and then dotting in the prompt frame for secondary position enhancement. Through appropriate Prompt prompt enhancement, SAM can segment the actually effective part of the information in the business opportunity picture.

[0110] In step S204, based on the prompt vector and the image feature vector, the initial image data is pre-segmented to generate a first pre-segmented image data corresponding to the text information region of the initial image data.

[0111] In an embodiment of the present application, based on the prompt vector and the image feature vector, the initial image data is pre-segmented to generate a segmentation mask; and the segmentation mask is optimized according to a preset mask optimization mode in the SAM network model, to output a first pre-segmented image data corresponding to the text information region of the initial image data.

[0112] In actual application, SAM uses a lightweight mask decoder (two layers of Transformer, 256 hidden units per layer) to generate high-quality segmentation masks according to image features and prompt information. Finally, the generated segmentation masks are optimized, including removing small objects and filling holes. Dice coefficient is used as the main evaluation index to evaluate the quality of the segmentation result, and the model is iteratively optimized according to the feedback.

[0113] In step S205, the first pre-segmented image data is input into the trained CRNN network model to output a first text detection result corresponding to the initial image data.

[0114] In the embodiment of the present application, the SAM algorithm can be constructed on the basis of a pre-trained visual transformer (ViT-H / 16 model) to extract image features (Feature Maps), learn to understand through prompt (such as points, boxes, and text) and predict a mask containing specific and effective information in a segmented business opportunity picture using a lightweight mask decoder combined with image encoding and prompt encoding. After this, the CRNN network is connected to the first pre-segmented image data output by the SAM, the image features are converted into feature sequences for processing, and the model CTC layer finally converts the label sequence output by the network into a final text string.

[0115] In the embodiment of the present application, initial image data to be processed can be obtained; the initial image data is input into a trained SAM network model to determine the image feature vector corresponding to the initial image data; prompt information corresponding to the SAM network model is obtained, and a prompt vector corresponding to the prompt information is determined; the initial image data is pre-segmented based on the prompt vector and the image feature vector to output first pre-segmented image data corresponding to the text information region of the initial image data, and the first pre-segmented image data is input into a trained CRNN network model to output a first text detection result corresponding to the initial image data, so as to realize the SAM+CRNN algorithm, pre-segmentation by the arbitrary segmentation performance of SAM, more accurate positioning of the text information part of the business opportunity picture, and text recognition by the CRNN network, so that the extraction of text information is more complete and accurate.

[0116] Referring to Figure 3 , a step flowchart of another information extraction method of image data provided by an embodiment of the present application is shown, which can specifically include the following steps:

[0117] Step S301, obtaining initial image data to be processed;

[0118] Step S302, inputting the initial image data into a trained SAM network model to output first pre-segmented image data corresponding to the text information region of the initial image data;

[0119] Step S303, inputting the pre-segmented image data into a trained CRNN network model, pre-processing the pre-segmented image data by using the CRNN network model to obtain first image data;

[0120] In an embodiment of the present application, the pre-segmented image data is converted into grayscale image data by inputting into a trained CRNN network model; binary image data is generated based on the grayscale image data; the binary image data is pre-processed according to a preset format specification to obtain the first image data.

[0121] In the CRNN network model, the CRNN image preprocessing can be realized, specifically, the color image segmented by the SAM can be converted into a gray image to reduce the complexity of processing data, and the threshold value is automatically determined by using the method of Otsu for binarization, so as to more clearly distinguish the text and background parts of the image; the median filter is applied to remove the noise in the image and enhance the text features; the Hough transform is used to detect the direction of the text line and correct the inclination.

[0122] After preprocessing, the CRNN network structure can be used for text recognition process and post-processing, specifically, the first image data can be adjusted to the same input size (256X256), and after normalization processing, the CNN feature extraction part uses the pre-trained ResNet-50 as the feature extractor to extract the deep features of the image. The LSTM which can effectively process long sequence data is used as the recurrent neural network in the RNN part, and finally the greedy decoding strategy is used to recover the text sequence from the CTC layer output, and the output index is mapped back to the corresponding character.

[0123] In the embodiment of the application, the text inclination scene can be processed: the OCR recognition algorithm CRNN can process various text distortions including inclination, and can effectively detect text information at different angles, while the traditional algorithm is usually for horizontal or vertical text content. The recognition adaptability is strong: it can adapt to pictures of different angles and sizes.

[0124] In step S304, the CNN structure in the CRNN network model is used to extract features from the first image data to obtain deep feature data.

[0125] In the CRNN network model, the CNN structure can realize feature extraction, after obtaining the first image data, the CNN structure can be used to realize feature extraction in one step to obtain deep feature data, and the deep feature data can reflect the text information in the image data, and then the text information extraction can be realized.

[0126] In step S305, the RNN structure in the CRNN network model is used to generate the first text detection result corresponding to the initial image data based on the deep feature data.

[0127] In the embodiment of the application, the initial image data to be processed is acquired; the initial image data is input into the trained SAM network model, first pre-segmented image data corresponding to the text information region of the initial image data is output, and the pre-segmented image data is input into the trained CRNN network model; the pre-segmented image data is preprocessed by using the CRNN network model to obtain first image data; the CNN structure in the CRNN network model is used to extract features of the first image data to obtain deep feature data; the RNN structure in the CRNN network model is used to generate a first text detection result corresponding to the initial image data based on the deep feature data, so that the SAM+CRNN algorithm is used, the pre-segmentation is performed by using the arbitrary segmentation performance of the SAM, the text information part of the business opportunity picture is more accurately positioned, and then the CRNN network is used for character recognition, so that the extraction of the text information is more complete and accurate.

[0128] Reference Figure 4 For the business opportunity picture processing method of the embodiment of the application, the following steps can be included:

[0129] Step S401, historical business opportunity pictures are acquired from an isee server;

[0130] Step S402, data labeling is performed on the acquired business opportunity pictures;

[0131] Step S403, data enhancement processing is performed;

[0132] Step S404, the image data processed by segmentation is segmented by using a SAM model;

[0133] Step S405, the SAM model is evaluated to determine whether an acceptance criterion is reached;

[0134] Step S406, when the SAM model reaches the acceptance criterion, the SAM model can be put into production application;

[0135] Step S407, when the SAM model does not reach the acceptance criterion, the SAM model is iteratively optimized;

[0136] Step S408, a business opportunity picture to be processed is acquired from the isee server;

[0137] Step S409, the business opportunity picture is segmented according to the SAM model;

[0138] Step S410, the segmented picture data is processed according to a preprocessing format specification;

[0139] Step S411, text recognition of the segmented picture data is performed by using a CRNN inference, and business opportunity text is output.

[0140] In practical applications, users upload pictures to the system, and the system automatically receives and prepares data for model processing.

[0141] 2. The model preprocesses the uploaded pictures, including conversion to grayscale images, binarization, denoising, etc., to improve the accuracy of recognition.

[0142] 3. The model inputs the processed pictures of uniform specifications, first through the basic SAM network architecture for pre-segmentation, and outputs the business opportunity picture text information region.

[0143] 4. Then the pre-segmented image is passed to CRNN, and the recognized sequence label is output using a CTC decoder to generate the final text line detection result.

[0144] In the embodiments of the present application, the SAM+CRNN network algorithm is used to realize the business opportunity picture information recognition and extraction, which has the following advantages:

[0145] Improving accuracy: By more accurately recognizing and positioning the rotating target, the overall accuracy of the picture recognition system can be improved, and the picture recognition accuracy of the iSee pen is above 90%.

[0146] Reducing labor costs: accurate automatic picture recognition reduces the pressure on manual review, shortens the business review time by 80%, and reduces the manual review amount by more than 90%.

[0147] It should be noted that for the method embodiments, in order to simply describe, they are expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited by the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0148] Referring to Figure 5 , a structural schematic diagram of an information extraction device for image data is shown, which can specifically include the following modules:

[0149] An initial image data acquisition module 501 is configured to acquire initial image data to be processed.

[0150] A first pre-segmented image data generation module 502 is configured to input the initial image data into a trained SAM network model and output first pre-segmented image data corresponding to the text information region of the initial image data.

[0151] The first text detection structure generation module 503 is configured to input the first pre-segmentation image data into a trained CRNN network model, and output a first text detection result corresponding to the initial image data.

[0152] In an embodiment of the present application, the first pre-segmentation image data generation module 502 can include:

[0153] An image feature vector determination sub-module is configured to input the initial image data into a trained SAM network model, and determine an image feature vector corresponding to the initial image data.

[0154] A prompt vector determination sub-module is configured to acquire prompt information corresponding to the SAM network model, and determine a prompt vector corresponding to the prompt information.

[0155] A first pre-segmentation image data output sub-module is configured to perform scene graph pre-segmentation on the initial image data based on the prompt vector and the image feature vector, and output first pre-segmentation image data corresponding to a text information region of the initial image data.

[0156] In an embodiment of the present application, the first pre-segmentation image data output sub-module can include:

[0157] A segmentation mask generation unit is configured to perform scene graph pre-segmentation on the initial image data based on the prompt vector and the image feature vector, and generate a segmentation mask.

[0158] A first pre-segmentation image data determination unit is configured to optimize the segmentation mask according to a preset mask optimization manner in the SAM network model, and output first pre-segmentation image data corresponding to a text information region of the initial image data.

[0159] In an embodiment of the present application, the first text detection structure generation module 503 can include:

[0160] A first image data determination sub-module is configured to input the pre-segmentation image data into a trained CRNN network model, and perform preprocessing on the pre-segmentation image data by using the CRNN network model to obtain first image data.

[0161] A deep feature data determination sub-module is configured to perform feature extraction on the first image data by using a CNN structure in the CRNN network model to obtain deep feature data.

[0162] A first text detection result generation module is configured to generate a first text detection result corresponding to the initial image data based on the deep feature data by using an RNN structure in the CRNN network model.

[0163] In an embodiment of the present application, the first image data determination sub-module can comprise:

[0164] a grayscale image data determination unit configured to input the trained CRNN network model and convert the pre-segmented image data into grayscale image data;

[0165] a binarization data generation unit configured to generate binarization image data based on the grayscale image data;

[0166] a first image data determination unit configured to pre-process the binarization image data according to a preset format specification to obtain first image data.

[0167] In an embodiment of the present application, the first image data determination unit can comprise:

[0168] a denoising processing sub-unit configured to perform denoising processing on the binarization image data to obtain first image data;

[0169] and / or a tilt correction sub-unit configured to detect the direction of a text line of the binarization image data and correct the tilt to obtain first image data;

[0170] and / or a normalization processing sub-unit configured to perform normalization processing on the binarization image data after adjusting the binarization image data to a preset input size to obtain first image data.

[0171] In an embodiment of the present application, the device further comprises:

[0172] a sample data set acquisition module configured to acquire a sample data set for training a network model, the sample data set comprising sample picture data and original text labels corresponding to the sample picture data;

[0173] a network structure construction module configured to construct an initial SAM network structure and an initial CRNN network structure;

[0174] a second pre-segmented image data module configured to input the sample picture data into the initial SAM network structure to obtain second pre-segmented image data;

[0175] a second text detection result output module configured to input the second pre-segmented image data and the original text labels into the initial CRNN network structure to output a second text detection result;

[0176] a loss function determination module configured to determine a loss function based on the second text detection result and the original text labels;

[0177] an iteration processing module configured to perform iteration on the initial SAM network structure and the initial CRNN network structure according to the loss function.

[0178] In the embodiment of the present application, the initial image data to be processed is acquired; the initial image data is input into the trained SAM network model, the first pre-segmentation image data corresponding to the text information region of the initial image data is output, and the first pre-segmentation image data is input into the trained CRNN network model, the first text detection result corresponding to the initial image data is output, so that the SAM+CRNN algorithm is used, the pre-segmentation is performed through the arbitrary segmentation performance of SAM, the text information part of the business opportunity picture is more accurately positioned, and then the CRNN network is used for character recognition, so that the extraction of the text information is more complete and accurate.

[0179] An embodiment of the present application further provides an electronic device, which can include a processor, a memory, and a computer program stored on the memory and capable of running on the processor, and the computer program is executed by the processor to realize the image data information extraction method.

[0180] An embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the image data information extraction method.

[0181] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0182] Each embodiment in the specification is described in a progressive manner, and each embodiment mainly describes the difference from other embodiments, and the same and similar parts of each embodiment refer to each other.

[0183] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0184] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0185] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices, such that a series of operational steps are performed on the computer or other programmable terminal devices to produce a computer implemented process so that the instructions which execute on the computer or other programmable terminal devices provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0187] Although preferred embodiments of the present application have been described, those skilled in the art will be able to make additional modifications and variations to the embodiments without departing from the scope of the present application. Accordingly, the appended claims are intended to encompass all such modifications and variations as falling within the scope of the embodiments of the present application.

[0188] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be intended to distinguish one entity or operation from another entity or operation without necessarily requiring or implying any such actual relationship or order between such entities or operations. Moreover, the terms "comprising", "comprising" or any other variations thereof are intended to cover non-exclusive inclusions, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or terminal device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or terminal device including the element.

[0189] The information extraction method and device for provided image data, electronic device, and storage medium are described in detail above, the principle and implementation manner of the present application are described by applying specific examples in the present article, the above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, according to the idea of the present application, the specific implementation manner and application range will have changes, and the above description should not be understood as the limitation of the present application.

Claims

1. A method for extracting information from image data, characterized in that, The method includes: Obtain the initial image data to be processed; The initial image data is input into the trained SAM network model, which outputs the first pre-segmented image data corresponding to the text information region in the initial image data. The first pre-segmented image data is input into the trained CRNN network model, which outputs the first text detection result corresponding to the initial image data.

2. The method according to claim 1, characterized in that, The step of inputting the initial image data into the trained SAM network model and outputting the first pre-segmented image data corresponding to the text information region in the initial image data includes: The initial image data is input into the trained SAM network model to determine the image feature vector corresponding to the initial image data; Obtain the prompt information corresponding to the SAM network model, and determine the prompt vector corresponding to the prompt information; Based on the prompt vector and the image feature vector, the initial image data is pre-segmented into a scene graph, and the first pre-segmented image data corresponding to the text information region in the initial image data is output.

3. The method according to claim 2, characterized in that, The step of pre-segmenting the initial image data based on the prompt vector and the image feature vector, and outputting the first pre-segmented image data corresponding to the text information region in the initial image data, includes: Based on the cue vector and the image feature vector, the initial image data is pre-segmented into a scene graph to generate a segmentation mask; The segmentation mask is optimized according to the preset mask optimization method in the SAM network model, and the first pre-segmented image data corresponding to the text information region in the initial image data is output.

4. The method according to claim 1, characterized in that, The step of inputting the pre-segmented image data into a trained CRNN network model and outputting the text detection result corresponding to the initial image data includes: The pre-segmented image data is input into a trained CRNN network model, and the CRNN network model is used to preprocess the pre-segmented image data to obtain the first image data. The CNN structure in the CRNN network model is used to extract features from the first image data to obtain deep feature data. The RNN structure in the CRNN network model is used to generate the first text detection result corresponding to the initial image data based on the deep feature data.

5. The method according to claim 4, characterized in that, The pre-segmented image data is input into a trained CRNN network model, and the CRNN network model is used to preprocess the pre-segmented image data to obtain first image data, including: The pre-segmented image data is converted into grayscale image data by inputting the trained CRNN network model. Binarized image data is generated based on the grayscale image data; The binarized image data is preprocessed according to a preset format specification to obtain the first image data.

6. The method according to claim 5, characterized in that, The preprocessing of the binarized image data according to a preset format specification to obtain the first image data includes: The binarized image data is denoised to obtain the first image data; And / or, detect the direction of the text lines in the binarized image data and correct the tilt to obtain the first image data; And / or, after adjusting the binarized image data to a preset input size, normalization processing is performed to obtain the first image data.

7. The method according to claim 1, characterized in that, Also includes: Obtain a sample dataset for training the network model, the sample dataset including sample image data and the original text labels corresponding to the sample image data; Construct the initial SAM network structure and the initial CRNN network structure; The sample image data is input into the initial SAM network structure to obtain the second pre-segmented image data; The second pre-segmented image data and the original text label are input into the initial CRNN network structure, and the second text detection result is output. The loss function is determined based on the second text detection result and the original text label; The initial SAM network structure and the initial CRNN network structure are iterated according to the loss function.

8. An image detection device, characterized in that, The device includes: The initial image data acquisition module is used to acquire the initial image data to be processed; The first pre-segmented image data generation module is used to input the initial image data into the trained SAM network model and output the first pre-segmented image data corresponding to the text information region in the initial image data. The first text detection structure generation module is used to input the first pre-segmented image data into a trained CRNN network model and output the first text detection result corresponding to the initial image data.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the information extraction method for image data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the information extraction method for image data as described in any one of claims 1 to 7.

Images