Image recognition method, device, equipment and storage medium
By determining the layout information of the ID image and using corresponding processing models for identification processing, the problem of low accuracy of ID image recognition in the existing technology is solved, and efficient image recognition effect is achieved and business processing efficiency is improved.
Patent Information
- Application Number
- CN202011462513.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-12-11
AI Technical Summary
Existing image recognition technology cannot perform targeted identification processing based on the layout format of the document, resulting in unsatisfactory recognition accuracy and it is difficult to meet the efficient business processing needs.
By obtaining the image to be identified of the target object, determining its layout information, and using the corresponding pre-trained structured positioning model or text detection model for identification based on the layout information, targeted processing is performed for images of different layout formats.
It significantly improves the accuracy of recognition, especially for images in structured layouts, which can accurately identify their structured information, improve business processing efficiency, and meet high-efficiency business needs.
Smart Images

Figure CN112580707B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image recognition method, device, equipment and storage medium. Background Art
[0002] With the continuous development of Internet technology, images have become one of the important information dissemination media. For example, when a user handles vehicle-related business, by providing the service provider with the image of the certificate such as the motor vehicle registration certificate, the service provider can use image recognition technology to identify and process the received certificate image to obtain the relevant information of the vehicle, and then process the business based on the relevant information of the vehicle without the need for manual input by the staff, thereby improving the efficiency of business processing.
[0003] In practical applications, many certificates include front and back sides with different layouts, such as motor vehicle registration certificates, property certificates, etc. However, in current image recognition technology, the same image recognition technology is usually used to identify and process the front and back sides of certificate images with different layouts, but targeted recognition cannot be performed based on the characteristics of the image. As a result, the recognition accuracy is not ideal, and the recognition results often need to be manually checked, which makes it difficult to truly meet the needs of efficient business processing. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an image recognition method, apparatus, device and storage medium to solve the problems that the current image recognition process cannot perform targeted recognition processing according to the characteristics of the image and the recognition accuracy is not ideal.
[0005] To solve the above technical problems, one or more embodiments of the present application are implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides an image recognition method, comprising:
[0007] Acquire an image of a target object to be identified; wherein the target object includes a first layout and a second layout with different layout formats;
[0008] Determining layout information of the target object included in the image to be identified;
[0009] If the layout information is the first layout, performing structured positioning processing on the image to be identified based on a pre-trained structured positioning model, and performing text recognition processing on the first target text to be identified obtained by the structured positioning processing to obtain first recognition result information;
[0010] If the layout information is the second layout, text detection processing is performed on the image to be identified based on a pre-trained text detection model, and text recognition processing is performed on the second target text to be identified obtained by the text detection processing to obtain second recognition result information.
[0011] In a second aspect, an embodiment of the present application provides an image recognition device, including:
[0012] An acquisition module, used for acquiring an image to be identified of a target object; wherein the target object includes a first layout and a second layout with different layout formats;
[0013] A determination module, used for determining the layout information of the target object included in the image to be identified;
[0014] A first recognition module is used for, if the layout information is the first layout, performing structured positioning processing on the image to be recognized based on a pre-trained structured positioning model, and performing text recognition processing on the first target text to be recognized obtained by the structured positioning processing to obtain first recognition result information;
[0015] The second recognition module is used to perform text detection processing on the image to be recognized based on a pre-trained text detection model if the layout information is the second layout, and perform text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information.
[0016] In a third aspect, an embodiment of the present application provides an image recognition device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the image recognition method provided in the first aspect above.
[0017] In a fourth aspect, an embodiment of the present application provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the image recognition method provided in the first aspect above.
[0018] The image recognition method, device, equipment and storage medium provided by the embodiments of the present application, when obtaining an image to be recognized including a target object of different layout formats, determines the layout information of the target object included in the image to be recognized, and uses a corresponding pre-trained processing model to perform recognition processing according to the determined layout information. Thus, for target objects with different layout formats, by pre-training processing models for images with different layouts, when obtaining an image to be recognized, different processing models can be used for targeted recognition processing according to the layout information of the target object included in the image to be recognized, rather than using the same recognition method for all images; therefore, the recognition accuracy is greatly improved, especially for images with structured layouts, the structured information can be accurately recognized, which is conducive to improving the processing efficiency of the business and can better meet the current high-efficiency business processing needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0020] Figure 1 A first flow chart of an image recognition method provided in an embodiment of the present application;
[0021] Figure 2 A second flow chart of an image recognition method provided in an embodiment of the present application;
[0022] Figure 3 A third flow chart of an image recognition method provided in an embodiment of the present application;
[0023] Figure 4 A fourth flow chart of an image recognition method provided in an embodiment of the present application;
[0024] Figure 5 A fifth flow chart of an image recognition method provided in an embodiment of the present application;
[0025] Figure 6 A sixth flow chart of an image recognition method provided in an embodiment of the present application;
[0026] Figure 7 A schematic diagram of the module composition of an image recognition device provided in an embodiment of the present application;
[0027] Figure 8A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0029] Figure 1 A flowchart of an image recognition method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:
[0030] Step S102, obtaining an image of a target object to be identified; wherein the target object includes a first layout and a second layout having different layout formats;
[0031] The executor of the image recognition method provided in the embodiment of the present application may be an image recognition device. Optionally, in the process of handling business, the user uses his terminal device to shoot the image to be recognized of the target object, and sends the image to be recognized to the image recognition device; the image recognition device receives the image to be recognized sent by the terminal device. Alternatively, in the process of handling business, the user uses his terminal device to shoot the image to be recognized of the target object, and sends the image to be recognized to the business processing system, and the business processing system sends the image to be recognized to the image recognition device; the image recognition device receives the image to be recognized sent by the business processing system. The manner in which the image recognition device obtains the image to be recognized of the target object is not specifically limited in this application, and it can be set by itself as needed in actual applications.
[0032] Among them, the target object can be a certificate, bill, etc. with a first layout and a second layout, such as a motor vehicle registration certificate, a real estate certificate, a student ID card, an invoice, a business voucher, etc. The first layout is a layout with a fixed style (such as a table style, etc.), multiple fixed fields, and no information can be added after generation, that is, it has a fixed structure, which is referred to as structured in this application; and each fixed field in the first layout is referred to as a structured field. In different target objects, each structured field has a unique corresponding text information, and in the recognition process, when a certain text information is located, the field information of the corresponding structured field can be determined. The second layout is a layout that does not have a fixed style and can add information. If it is a blank layout, information can be added anywhere; or a layout with a fixed style and can add information, such as a table structure, in which the first row or each column does not have a unique corresponding text information, which varies with the entity corresponding to the target object or the business or operation in which the target object itself participates. Taking the target object as the motor vehicle registration certificate and the entity corresponding to the target object as vehicle 1 as an example, the first page includes a table style, which includes structured fields such as the motor vehicle owner, registration authority, registration time, registration number, registration certificate number, vehicle type, vehicle identification code, engine model, displacement and power; among them, the only text information corresponding to the structured field "motor vehicle owner" is "user name". During the recognition process, when the user name "Zhang San" is located, it can be determined that the field information of the corresponding structured field is the vehicle owner. The same is true for other structured fields, which will not be explained one by one here; the second page of the motor vehicle registration certificate includes a table with multiple rows. When vehicle 1 has participated in a pledge business, one of the rows contains corresponding pledge information; when vehicle 1 has participated in a buying and selling business, one of the rows contains corresponding buying and selling information; when vehicle 1 has participated in any business, each row is blank.
[0033] Step S104, determining the layout information of the target object included in the image to be identified;
[0034] Specifically, Figure 2 As shown, step S104 may include the following steps S104-2 and S104-4:
[0035] Step S104-2, classifying the image to be identified according to the pre-trained classification model to obtain classification result information of the image to be identified;
[0036] Specifically, the image to be identified is input into a pre-trained classification model for classification processing to obtain classification result information of the image to be identified. Considering that in actual applications, due to different shooting angles, the orientation of the target object in the image to be identified is often different, in order to facilitate subsequent identification processing, in one or more embodiments of the present application, the classification result information may include a field representing the layout information and a field representing the orientation information of the target object in the image to be identified.
[0037] Step S104-4, determining the layout information of the target object included in the image to be identified according to the classification result information.
[0038] Specifically, a field representing the layout information is obtained from the classification result, and the layout information of the target object included in the image to be identified is determined according to the obtained field. As an example, the format of the classification result information is "layout_angle". When the field layout is front, the representation is the first layout, and when the field layout is back, the representation is the second layout; when the angle is 0, the orientation of the target object in the image to be identified is the preset orientation, and when the angle is other values, the orientation of the target object in the image to be identified is represented by the angle of rotation relative to the preset orientation. For example, if the classification result information is front_90, and the field representing the layout information obtained from the classification result information is front, it is determined that it is the first layout, and the orientation information obtained is 90, it is determined that the target object is rotated 90 degrees clockwise relative to the preset orientation in the image to be identified. It should be pointed out that the classification result information is not limited to the above format, and it can be set according to needs in actual applications.
[0039] Furthermore, in order to accurately and effectively locate the text and perform text recognition in the subsequent recognition process, when it is determined according to the classification result information that the target object is not in the preset position in the image to be recognized, the image to be recognized can also be corrected. Figure 3 As shown, after step S104-4, the following steps S104-6 and S104-8 may also be included:
[0040] Step S104-6, determining the position information of the target object in the image to be identified according to the classification result information;
[0041] Step S104-8: If it is determined according to the position information that the target object is not in the preset position in the image to be identified, correction processing is performed on the image to be identified.
[0042] For example, if it is determined based on the orientation information that the target object in the image to be identified is rotated 90 degrees clockwise relative to the preset orientation, the image to be identified is rotated 90 degrees counterclockwise so that the target object is in the preset orientation in the image to be identified; wherein the preset orientation can be set as needed in actual applications.
[0043] Corresponding to step S104-6 and step S106-8, Figure 3 As shown, step S106 may include the following steps S1062, and step S108 may include the following steps S1082:
[0044] Step S1062: if the determined layout information is the first layout, then a structured positioning process is performed on the image to be recognized after the correction process based on a pre-trained structured positioning model, and a text recognition process is performed on the first target text to be recognized obtained by the structured positioning process to obtain first recognition result information;
[0045] Step S1082, if the determined layout information is the second layout, text detection processing is performed on the corrected image to be identified based on a pre-trained text detection model, and text recognition processing is performed on the second target text to be identified obtained by the text detection processing to obtain second recognition result information.
[0046] Furthermore, in order to accurately determine the layout information and the orientation information, in one or more embodiments of the present application, a classification model training process is performed in advance. Specifically, before step S104, the following steps A2 to A6 may also be included:
[0047] Step A2, obtaining a first sample image of a plurality of target object samples;
[0048] Optionally, the first sample images of multiple target object samples are obtained from the network; or the first sample images of multiple target object samples are obtained from a specified image library, wherein the image library may be an image library of an authoritative organization, or a database constructed based on images of target objects provided by users during business handling. The method for obtaining the first sample image may be set as required in actual applications. As an example, the target object is a motor vehicle registration certificate, and the target object sample is a motor vehicle registration certificate sample.
[0049] It should be noted that, for the same target object sample, one first sample image may be acquired, or a plurality of first sample images containing target object samples in different orientations or layouts may be acquired.
[0050] Step A4, performing a labeling process on each first sample image to label the layout information and orientation information of the target object sample in the first sample image;
[0051] Considering that in actual applications, a small tilt of the target object relative to the preset orientation in the image to be identified has little effect on the identification of information, and when the user takes an image of the target object, the target object is often rotated 90 degrees, 180 degrees, 270 degrees, etc. relative to the preset orientation. Based on this, in one or more embodiments of the present application, 8 categories are pre-set, namely, the first layout and in a preset orientation (which can be recorded as title_front_0), the first layout and rotated 90 degrees clockwise relative to the preset orientation (which can be recorded as title_front_90), the first layout and rotated 180 degrees clockwise relative to the preset orientation (which can be recorded as title_front_180), the first layout and rotated 270 degrees clockwise relative to the preset orientation (which can be recorded as title_front_270), the second layout and in a preset orientation (which can be recorded as title_back_0), the second layout and rotated 90 degrees clockwise relative to the preset orientation (which can be recorded as title_back_90), the second layout and rotated 180 degrees clockwise relative to the preset orientation (which can be recorded as title_back_180), and the second layout and rotated 270 degrees clockwise relative to the preset orientation (which can be recorded as title_back_270). And the corresponding layout information and orientation information are marked according to the category to which the first sample image belongs.
[0052] Step A6: According to a preset first training method, training is performed based on the labeled first sample image to obtain a classification model.
[0053] Specifically, the first sample image after the annotation processing is divided into a first training set and a first test set according to a preset ratio; the first training set is trained according to a preset first training method to obtain an initial classification model; the initial classification model is tested using the first test set, and if the test result meets the preset conditions, the corresponding initial classification model is determined as the final classification model; if the test result does not meet the preset conditions, the training process is re-performed based on the first training set until the final classification model is obtained. Among them, the first training method can be set by itself as needed in actual applications, for example, based on the VGG16 network, the training process is performed using the stochastic gradient descent method and the cross entropy loss function. Since the training process and the testing process of the model are technical means well known to those skilled in the art, the training process and the testing process of the model will not be further described in detail in this application.
[0054] Therefore, by pre-training the classification model and when obtaining the image to be identified of the target object, determining the layout information and orientation information of the target object in the image to be identified based on the classification model, not only can the corresponding recognition processing model be determined according to the layout information, thereby performing accurate information recognition processing; but also the image to be identified can be corrected according to the orientation information, thereby improving the accuracy of subsequent recognition processing.
[0055] Step S106, if the layout information is the first layout, then performing structured positioning processing on the image to be recognized based on the pre-trained structured positioning model, and performing text recognition processing on the first target text to be recognized obtained by the structured positioning processing to obtain first recognition result information;
[0056] Specifically, Figure 4 As shown, step S106 may include the following steps S106-2 to S106-6:
[0057] Step S106-2: if the layout information is the first layout, the image to be identified is input into a pre-trained structured positioning model for structured positioning processing to obtain a positioning frame of the first target text to be identified and field information of a structured field corresponding to the first target text;
[0058] It should be noted that when the image to be recognized is corrected in step S104, the corrected image to be recognized is input into the pre-trained structured positioning model for structured positioning processing to obtain the positioning frame of the first target text to be recognized and the field information of the structured field corresponding to the first target text. The first target text may include any one or more combinations of text, numbers, letters, etc.
[0059] Since the first layout of the target object has a fixed style and structure; therefore, it can be understood that the structured positioning model in the embodiment of the present application can locate the first target text to be identified corresponding to each structured field in the table structure, and determine the field information of the structured field corresponding to the first target text. For example, the target object is a motor vehicle registration certificate, and its first layout includes a structured field "vehicle type" and corresponding text information "small ordinary passenger car". When the image recognition device obtains the image to be identified of the motor vehicle registration certificate, the first target text to be identified obtained by positioning processing based on the structured positioning model includes "small ordinary passenger car", a positioning box is added to it, and the field information of the structured field corresponding to the target text "small ordinary passenger car" can be determined to be "vehicle type".
[0060] Step S106 - 4 , performing image extraction processing according to the positioning frame to obtain a first sub-image including the first target text.
[0061] Specifically, image extraction processing is performed on the area enclosed by each positioning frame to obtain a corresponding first sub-image including the first target text.
[0062] Step S106 - 6 , inputting the first sub-image and the corresponding field information into a pre-trained text recognition model for text recognition processing to obtain first recognition result information.
[0063] Among them, the first recognition result information includes field information and the corresponding text information obtained by recognition; for example, the text information obtained by recognizing and processing the above-mentioned first target text "small ordinary passenger car" is a small ordinary passenger car, and the first recognition result information includes "vehicle type: small ordinary passenger car". Therefore, the business processing system can automatically enter small ordinary passenger car in the vehicle type column on the business processing page based on the first recognition result information obtained by the image recognition device, without the need for manual input by the staff, thereby greatly improving the business processing efficiency. It should be pointed out that the format of the first recognition result information is not limited to the above format, and can be set according to needs in actual applications.
[0064] Furthermore, in order to avoid the image being too large or too small and affecting the accuracy of recognition, in one or more embodiments of the present application, the first sub-image can be scaled according to a preset aspect ratio to obtain a first sub-image of a preset size, and then the first sub-image of the preset size and the corresponding field information can be input into a pre-trained text recognition model for text recognition processing to obtain first recognition result information.
[0065] Considering that many certificates or bills are currently formulated according to industry regulations, different certificates or bills of the same type have a unified layout, and each structured field has a strong positionality. In order to accurately identify the text information corresponding to each structured field in the first layout of the target object included in the image to be identified, in one or more embodiments of the present application, a structured positioning model is pre-trained. Specifically, before step S106-2, the following steps B2 to B6 may also be included:
[0066] Step B2, obtaining a third sample image of a plurality of target object samples; wherein the third sample image includes a first layout of the target object samples;
[0067] The method for obtaining the third sample image may refer to the method for obtaining the first sample image, which is not specifically limited in the present application and may be set as required in actual applications.
[0068] Step B4, performing a labeling process on each third sample image to label the structured fields in the first layout;
[0069] Taking the target object as a motor vehicle registration certificate as an example, pixels and field information can be annotated for each structured field included in the first layout, such as the vehicle owner, registration authority, registration time, registration number, registration certificate number, vehicle type, vehicle identification code, engine model, engine number, vehicle brand, vehicle manufacturer, number of passengers, displacement and power in each third sample image.
[0070] Step B6: Perform training processing based on the third sample image after labeling according to a preset third training method to obtain a structured positioning model.
[0071] Considering that in practical applications, different images to be identified are often taken in different environments, and their sizes, angles, colors, etc. are different. In order to enable the structured positioning model to take into account a variety of images to be identified and have better robustness, in one or more embodiments of the present application, the third sample image after the annotation processing is firstly subjected to data enhancement processing, such as random cropping, random scaling, random flipping, affine transformation, color jittering, mosaic enhancement, adaptive anchor calculation, etc. The third sample image after the data enhancement processing is divided into a third training set and a third test set according to a preset ratio, and the third training set is trained according to the preset third training method to obtain an initial structured positioning model; the initial structured positioning model is tested using the third test set, and when the test result meets the preset conditions, the corresponding initial structured positioning model is determined as the final structured positioning model; when the test result does not meet the preset conditions, the training process is re-performed based on the third training set until the final structured positioning model is obtained. Among them, the third training method can be set by itself as needed in actual applications. For example, the third training set is trained based on the target detection framework YOLOv5. In the process of training the training set based on the target detection framework YOLOv5, feature extraction can be completed based on the Focus and CSP structures, wherein the Focus and CSP structures are a special network structure that can increase feature combinations and prevent gradient disappearance; and the Neck structure composed of FPN and PAN is used to enhance feature fusion and realize detection of targets of different scales and sizes; and the objective function is composed of GIOU_Loss and Boundingbox loss functions, and the Adam optimizer is used to optimize the training to ensure the accuracy of the trained structured positioning model.
[0072] Furthermore, considering that although certificates, bills, etc. include multiple fields, each field is usually horizontal text, in order to improve text recognition efficiency and reduce computing overhead, in one or more embodiments of the present application, a text recognition model is pre-trained, and recognition processing is performed on the target text to be recognized based on the text recognition model. Specifically, the following steps C2 and C4 may also be included before step S106-6:
[0073] Step C2, obtaining second sample images of multiple target object samples, and performing a labeling process on each second sample image to label the text information to be identified;
[0074] Since the training of recognition models often requires millions of training data, relying solely on manual labeling will require a lot of manpower and time costs; therefore, a part of the second sample images with labeled information can be obtained through manual labeling, a part of the second sample images with labeled information can be obtained from a public image library, and a part of the second sample images with labeled information can be simulated and generated through computer image processing technology.
[0075] Step C4, according to a preset second training method, training is performed based on the second sample image after the annotation process to obtain a text recognition model.
[0076] Considering that the second sample images are often of different sizes, in order to improve the accuracy of the text recognition model and avoid severe deformation of the image, in one or more embodiments of the present application, each annotated second sample image is scaled according to a preset aspect ratio to obtain a second sample image with labeling information of a preset size. The sample image with labeling information of the preset size is divided into a second training set and a second test set according to a preset ratio; the second training set is trained based on a preset second training method to obtain an initial text recognition model; the initial text recognition model is tested according to the second test set, and when the test result meets the preset conditions, the initial text recognition model is determined as the final text recognition model; when the test result does not meet the preset conditions, the training is re-performed according to the second training set until the final text recognition model is obtained. Among them, the second training method can be set according to actual needs in actual applications, such as training based on a CRNN network, an Adadelta optimizer, and a loss function CTC-Loss (Connectionist Temporal Classification).
[0077] By training the text recognition model, text recognition processing is performed based on the text recognition model, rather than the traditional method of establishing a specific character recognition model based on a specific field. This greatly reduces the computing overhead and improves the recognition efficiency.
[0078] Step S108, if the layout information is the second layout, text detection processing is performed on the image to be recognized based on a pre-trained text detection model, and text recognition processing is performed on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information.
[0079] Specifically, Figure 5 As shown, step S108 may include the following steps S108-2 to S108-6:
[0080] Step S108-2: if the layout information is the second layout, the image to be identified is input into a pre-trained text detection model for text detection processing to obtain a text area;
[0081] Step S108-4, performing image extraction processing according to the text area to obtain a second sub-image including a second target text to be recognized;
[0082] Step S108-6: input the second sub-image into a pre-trained text recognition model for text recognition processing to obtain second recognition result information.
[0083] The text recognition model is the same as the text recognition model in step S106 , and the related description can refer to the above description.
[0084] Since the formats of the first and second layouts of the target object are different, and the second layout does not have a unified structured style, in order to accurately detect the text in the second layout, thereby performing text recognition; in one or more embodiments of the present application, a text detection model is pre-trained. Specifically, before step S108, the following steps D2 to D6 are also included:
[0085] Step D2, obtaining a fourth sample image of a plurality of target object samples; wherein the fourth sample image includes a second layout of the target object;
[0086] The method for obtaining the fourth sample image may refer to the method for obtaining the first sample image, and this application does not make any specific limitation thereto.
[0087] Step D4, performing labeling processing on each fourth sample image to label the text area;
[0088] Step D6, according to a preset fourth training method, training is performed based on the fourth sample image after the annotation process to obtain a text detection model.
[0089] Specifically, the fourth sample image is divided into a fourth training set and a fourth test set according to a preset ratio, and the fourth training set is trained according to a preset fourth training method to obtain an initial text detection model; the initial text detection model is tested according to the fourth test set to obtain test result information, and if it is determined that the test result information meets the preset conditions, the initial text detection model is determined as the final text detection model, and if it is determined that the test result information does not meet the preset conditions, the training process is re-performed according to the fourth training set until the final text detection model is obtained. Among them, the fourth training method can be set according to actual needs in actual applications, such as training based on a ResNeSt network.
[0090] Therefore, based on the uncertainty of the text information in the second layout, by pre-training a text detection model, and first performing text detection processing on the image to be identified including the second layout based on the text detection model, and then performing text recognition processing on the second target text obtained by detection, effective recognition of the text information of the second layout is achieved.
[0091] Currently, OCR (Optical Character Recognition) technology is a widely used image recognition technology. Considering that in the OCR recognition process, multiple pre-processing operations such as binarization, uneven illumination correction, noise removal, and character segmentation are required for the image, not only the operation is cumbersome, but also the recognition result is easily affected by the image quality. Based on this, in this application, the corresponding processing model is pre-trained for images including different layouts, and the corresponding processing is performed based on the model. Image recognition can be achieved with only simple image pre-processing or without image pre-processing, which not only improves the recognition efficiency, but also reduces the impact of pre-processing on the recognition result.
[0092] Furthermore, considering that the second page of the target object usually includes relevant information of the entity corresponding to the target object or the business operation in which the target object participates; for example, if the target object is a motor vehicle registration certificate, its second page usually includes information of the corresponding vehicle participating in business operations such as pledge, sale, etc. For another example, if the target object is a student ID card, its second page usually includes information of ticket purchase operations performed by the student ID card, etc. In order to facilitate the business processing system to obtain the number of operations of the entity corresponding to the target object or the target object for a specified operation during the business processing process, in one or more embodiments of the present application, such as Figure 6 As shown, after step S108, the following step S110 may be further included:
[0093] Step S110: determining the number of times the entity corresponding to the target object or the target object performs a designated operation according to the second recognition result information.
[0094] Specifically, the similarity between each text information in the second recognition result information and each standard text in the specified standard text library is determined according to a preset method; if it is determined that the similarity is greater than a preset similarity threshold, the operation type information corresponding to the text information is determined according to the corresponding standard text; according to the determined operation type information, the number of operations of the entity or target object corresponding to the target object to perform the specified operation is determined. More specifically, an association relationship between each standard text and the operation type information is established in advance, and when it is determined that the similarity is greater than a preset similarity threshold, the associated operation type information is obtained from the association relationship according to the corresponding standard text; the number of identical operation type information is counted, and the counted number is determined as the number of operations of the corresponding operation. Among them, the preset method is such as calculating the cosine distance. As an example, the target object is a motor vehicle registration certificate, and the second recognition result information includes text information: transferred from Zhang to Li on September 1, 2010, and transferred from Li to Wang on August 6, 2015; after calculating the similarity between each text information and the standard text, it is determined that the corresponding operation type information is a transfer type, so the number of transfers of the corresponding vehicle can be obtained by statistics as 2 times.
[0095] Therefore, for the images to be identified including the second layout, after text recognition processing, by determining the entity corresponding to the target object or the number of operations of the specified operation performed by the target object, it is possible to provide the business system with valid business data without manual statistics, which not only improves business processing efficiency but also avoids statistical errors caused by human errors.
[0096] In one or more embodiments of the present application, when an image to be identified of a target object with different layout formats is obtained, the layout information of the target object included in the image to be identified is determined, and a corresponding pre-trained processing model is used to perform identification processing according to the determined layout information to obtain identification result information. Thus, for target objects with different layout formats, by pre-training processing models for images with different layouts, when an image to be identified is obtained, different processing models can be used to perform targeted identification processing according to the layout information of the target object included in the image to be identified, rather than using the same identification method for all images; therefore, the accuracy of identification is greatly improved, especially for images with structured layouts, the structured information can be accurately identified, which is conducive to improving the processing efficiency of the business and can better meet the current high-efficiency business processing needs.
[0097] Furthermore, corresponding to the image recognition method described above, based on the same technical concept, an embodiment of the present application also provides an image recognition device. Figure 7 A schematic diagram of the module composition of an image recognition device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device comprises:
[0098] An acquisition module 201 is used to acquire an image to be recognized of a target object; wherein the target object includes a first layout and a second layout of different layout formats;
[0099] A determination module 202, configured to determine the layout information of the target object included in the image to be identified;
[0100] The first recognition module 203 is used for, if the layout information is the first layout, performing structured positioning processing on the image to be recognized based on a pre-trained structured positioning model, and performing text recognition processing on the first target text to be recognized obtained by the structured positioning processing to obtain first recognition result information;
[0101] The second recognition module 204 is used to perform text detection processing on the image to be recognized based on a pre-trained text detection model if the layout information is the second layout, and perform text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information.
[0102] Optionally, the determination module 202 classifies the image to be identified according to a pre-trained classification model to obtain classification result information of the image to be identified;
[0103] The layout information of the target object included in the image to be identified is determined according to the classification result information.
[0104] Optionally, the device further comprises: a correction module;
[0105] The correction module determines the position information of the target object in the image to be identified according to the classification result information;
[0106] If it is determined according to the position information that the target object is not in a preset position in the image to be identified, correction processing is performed on the image to be identified.
[0107] Optionally, the device further comprises: a first training module;
[0108] The first training module obtains a first sample image of a plurality of target object samples;
[0109] Performing a labeling process on each of the first sample images to label the layout information and orientation information of the target object sample in the first sample image;
[0110] According to a preset first training method, training processing is performed based on the first sample image after labeling to obtain the classification model.
[0111] Optionally, the first recognition module 203 inputs the image to be recognized into a pre-trained structured positioning model for structured positioning processing to obtain a positioning frame of the first target text to be recognized and field information of a structured field corresponding to the first target text;
[0112] Performing image extraction processing according to the positioning frame to obtain a first sub-image including the first target text;
[0113] The first sub-image and the corresponding field information are input into a pre-trained text recognition model for text recognition processing to obtain first recognition result information.
[0114] Optionally, the second recognition module 204 inputs the image to be recognized into a pre-trained text detection model to perform text detection processing to obtain a text area;
[0115] Performing image extraction processing according to the text area to obtain a second sub-image including the second target text to be recognized;
[0116] The performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information includes:
[0117] The second sub-image is input into a pre-trained text recognition model for text recognition processing to obtain second recognition result information.
[0118] Optionally, the device further comprises: a statistical module;
[0119] The statistical module determines the entity corresponding to the target object or the number of operations of the target object performing a specified operation according to the second recognition result information after the second recognition module 204 performs text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information.
[0120] Optionally, the statistical module determines the similarity between each text information in the second recognition result information and each standard text in a specified standard text library according to a preset method;
[0121] If it is determined that the similarity is greater than a preset similarity threshold, determining the operation type information corresponding to the text information according to the corresponding standard text;
[0122] According to the determined operation type information, the entity corresponding to the target object or the number of operations of the target object performing the specified operation is determined.
[0123] Optionally, the device further comprises: a second training module;
[0124] The second training module obtains second sample images of multiple target object samples;
[0125] Performing a labeling process on each of the second sample images to label the text information to be recognized;
[0126] According to a preset second training method, training is performed based on the second sample image after annotation to obtain the text recognition model.
[0127] Optionally, the device further comprises: a third training module;
[0128] The third training module obtains a third sample image of a plurality of target object samples; wherein the third sample image includes a first layout of the target object sample;
[0129] Performing a labeling process on each of the third sample images to label the structured fields in the first layout;
[0130] According to a preset third training method, training processing is performed based on the third sample image after labeling processing to obtain the structured positioning model.
[0131] Optionally, the device further comprises: a fourth training module;
[0132] The fourth training module obtains a fourth sample image of a plurality of target object samples; wherein the fourth sample image includes a second layout of the target object;
[0133] Performing a labeling process on each of the fourth sample images to label a text area;
[0134] According to the preset fourth training method, training processing is performed based on the fourth sample image after annotation processing to obtain the text detection model.
[0135] The image recognition device provided by one or more embodiments of the present application determines the layout information of the target object included in the image to be recognized when acquiring an image to be recognized including a target object of different layout formats, and uses a corresponding pre-trained processing model to perform recognition processing according to the determined layout information. Thus, for target objects with different layout formats, by pre-training processing models for images with different layouts, when the image to be recognized is acquired, different processing models can be used to perform targeted recognition processing according to the layout information of the target object included in the image to be recognized, rather than using the same recognition method for all images; therefore, the recognition accuracy is greatly improved, especially for images with structured layouts, the structured information can be accurately recognized, which is conducive to improving the processing efficiency of the business and can better meet the current high-efficiency business processing needs.
[0136] It should be noted that the embodiment of the image recognition device in the present application and the embodiment of the image recognition method in the present application are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding image recognition method mentioned above, and the repeated parts will not be repeated.
[0137] Furthermore, corresponding to the above-mentioned image recognition method, based on the same technical concept, an embodiment of the present application further provides an image recognition device, which is used to execute the above-mentioned image recognition method. Figure 8 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application.
[0138] like Figure 8 As shown, the image recognition device may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and memory 302, and the memory 302 may store one or more storage applications or data. Among them, the memory 302 may be a short-term storage or a permanent storage. The application stored in the memory 302 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the image recognition device. Furthermore, the processor 301 may be configured to communicate with the memory 302, and execute a series of computer executable instructions in the memory 302 on the image recognition device. The image recognition device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more input and output interfaces 305, one or more keyboards 306, etc.
[0139] In a specific embodiment, the image recognition device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the image recognition device, and the one or more programs configured to be executed by one or more processors include computer executable instructions for performing the following:
[0140] Acquire an image of a target object to be identified; wherein the target object includes a first layout and a second layout with different layout formats;
[0141] Determining layout information of the target object included in the image to be identified;
[0142] If the layout information is the first layout, performing structured positioning processing on the image to be identified based on a pre-trained structured positioning model, and performing text recognition processing on the first target text to be identified obtained by the structured positioning processing to obtain first recognition result information;
[0143] If the layout information is the second layout, text detection processing is performed on the image to be identified based on a pre-trained text detection model, and text recognition processing is performed on the second target text to be identified obtained by the text detection processing to obtain second recognition result information.
[0144] Optionally, when the computer executable instruction is executed, determining the layout information of the target object included in the image to be identified includes:
[0145] Classify the image to be identified according to a pre-trained classification model to obtain classification result information of the image to be identified;
[0146] The layout information of the target object included in the image to be identified is determined according to the classification result information.
[0147] Optionally, when the computer executable instructions are executed, the method further comprises:
[0148] Determine the position information of the target object in the image to be identified according to the classification result information;
[0149] If it is determined according to the position information that the target object is not in a preset position in the image to be identified, correction processing is performed on the image to be identified.
[0150] Optionally, when the computer executable instructions are executed, the method further includes:
[0151] Acquire a first sample image of a plurality of target object samples;
[0152] Performing a labeling process on each of the first sample images to label the layout information and orientation information of the target object sample in the first sample image;
[0153] According to a preset first training method, training processing is performed based on the first sample image after labeling to obtain the classification model.
[0154] Optionally, when the computer executable instructions are executed, the performing structured positioning processing on the image to be identified based on the pre-trained structured positioning model includes:
[0155] Inputting the image to be recognized into a pre-trained structured positioning model for structured positioning processing to obtain a positioning frame of the first target text to be recognized and field information of a structured field corresponding to the first target text;
[0156] Performing image extraction processing according to the positioning frame to obtain a first sub-image including the first target text;
[0157] The performing text recognition processing on the first target text to be recognized obtained by the structured positioning processing to obtain first recognition result information includes:
[0158] The first sub-image and the corresponding field information are input into a pre-trained text recognition model for text recognition processing to obtain first recognition result information.
[0159] Optionally, when the computer executable instructions are executed, text detection processing is performed on the image to be recognized based on a pre-trained text detection model, including:
[0160] Inputting the image to be recognized into a pre-trained text detection model to perform text detection processing to obtain a text area;
[0161] Performing image extraction processing according to the text area to obtain a second sub-image including the second target text to be recognized;
[0162] The performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information includes:
[0163] The second sub-image is input into a pre-trained text recognition model for text recognition processing to obtain second recognition result information.
[0164] Optionally, when the computer executable instructions are executed, after performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information, the instructions further include:
[0165] The entity corresponding to the target object or the number of operations of the target object performing a designated operation is determined according to the second recognition result information.
[0166] Optionally, when the computer executable instructions are executed, determining the entity corresponding to the target object or the number of operations of the target object performing a specified operation according to the second recognition result information includes:
[0167] Determine the similarity between each text information in the second recognition result information and each standard text in the specified standard text library according to a preset method;
[0168] If it is determined that the similarity is greater than a preset similarity threshold, determining the operation type information corresponding to the text information according to the corresponding standard text;
[0169] According to the determined operation type information, the entity corresponding to the target object or the number of operations of the target object performing the specified operation is determined.
[0170] Optionally, when the computer executable instructions are executed, the method further includes:
[0171] Acquire a second sample image of a plurality of target object samples;
[0172] Performing a labeling process on each of the second sample images to label the text information to be recognized;
[0173] According to a preset second training method, training is performed based on the second sample image after annotation to obtain the text recognition model.
[0174] Optionally, when the computer executable instructions are executed, the method further includes:
[0175] Acquire a third sample image of a plurality of target object samples; wherein the third sample image includes a first layout of the target object samples;
[0176] Performing a labeling process on each of the third sample images to label the structured fields in the first layout;
[0177] According to a preset third training method, training processing is performed based on the third sample image after labeling processing to obtain the structured positioning model.
[0178] Optionally, when the computer executable instructions are executed, the method further includes:
[0179] Acquire a fourth sample image of a plurality of target object samples; wherein the fourth sample image includes a second layout of the target object;
[0180] Performing a labeling process on each of the fourth sample images to label a text area;
[0181] According to the preset fourth training method, training processing is performed based on the fourth sample image after annotation processing to obtain the text detection model.
[0182] The image recognition device provided in the embodiment of the present application, when acquiring an image to be recognized including a target object of different layout formats, determines the layout information of the target object included in the image to be recognized, and uses a corresponding pre-trained processing model to perform recognition processing according to the determined layout information. Thus, for target objects with different layout formats, by pre-training processing models for images with different layouts, when acquiring an image to be recognized, it is possible to use different processing models to perform targeted recognition processing according to the layout information of the target object included in the image to be recognized, rather than using the same recognition method for all images; therefore, the recognition accuracy is greatly improved, especially for images with structured layouts, the structured information can be accurately recognized, which is conducive to improving the processing efficiency of the business and can better meet the current high-efficiency business processing needs.
[0183] It should be noted that the embodiment of the image recognition device in the present application and the embodiment of the image recognition method in the present application are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding image recognition method mentioned above, and the repeated parts will not be repeated.
[0184] Further, corresponding to the above data processing method, based on the same technical concept, one or more embodiments of the present application further provide a storage medium for storing computer executable instructions. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instructions stored in the storage medium are executed by the processor, the following process can be implemented:
[0185] Acquire an image of a target object to be identified; wherein the target object includes a first layout and a second layout with different layout formats;
[0186] Determining layout information of the target object included in the image to be identified;
[0187] If the layout information is the first layout, performing structured positioning processing on the image to be identified based on a pre-trained structured positioning model, and performing text recognition processing on the first target text to be identified obtained by the structured positioning processing to obtain first recognition result information;
[0188] If the layout information is the second layout, text detection processing is performed on the image to be identified based on a pre-trained text detection model, and text recognition processing is performed on the second target text to be identified obtained by the text detection processing to obtain second recognition result information.
[0189] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, the determining of the layout information of the target object included in the image to be identified includes:
[0190] Classify the image to be identified according to a pre-trained classification model to obtain classification result information of the image to be identified;
[0191] The layout information of the target object included in the image to be identified is determined according to the classification result information.
[0192] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, the method further includes:
[0193] Determine the position information of the target object in the image to be identified according to the classification result information;
[0194] If it is determined according to the position information that the target object is not in a preset position in the image to be identified, correction processing is performed on the image to be identified.
[0195] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, the method further includes:
[0196] Acquire a first sample image of a plurality of target object samples;
[0197] Performing a labeling process on each of the first sample images to label the layout information and orientation information of the target object sample in the first sample image;
[0198] According to a preset first training method, training processing is performed based on the first sample image after labeling to obtain the classification model.
[0199] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, the structured positioning processing is performed on the image to be identified based on the pre-trained structured positioning model, including:
[0200] Inputting the image to be recognized into a pre-trained structured positioning model for structured positioning processing to obtain a positioning frame of the first target text to be recognized and field information of a structured field corresponding to the first target text;
[0201] Performing image extraction processing according to the positioning frame to obtain a first sub-image including the first target text;
[0202] The performing text recognition processing on the first target text to be recognized obtained by the structured positioning processing to obtain first recognition result information includes:
[0203] The first sub-image and the corresponding field information are input into a pre-trained text recognition model for text recognition processing to obtain first recognition result information.
[0204] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, text detection processing is performed on the image to be recognized based on a pre-trained text detection model, including:
[0205] Inputting the image to be recognized into a pre-trained text detection model to perform text detection processing to obtain a text area;
[0206] Performing image extraction processing according to the text area to obtain a second sub-image including the second target text to be recognized;
[0207] The performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information includes:
[0208] The second sub-image is input into a pre-trained text recognition model for text recognition processing to obtain second recognition result information.
[0209] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, after performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information, the method further includes:
[0210] The entity corresponding to the target object or the number of operations of the target object performing a designated operation is determined according to the second recognition result information.
[0211] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, determining the entity corresponding to the target object or the number of operations of the target object performing a specified operation according to the second recognition result information includes:
[0212] Determine the similarity between each text information in the second recognition result information and each standard text in the specified standard text library according to a preset method;
[0213] If it is determined that the similarity is greater than a preset similarity threshold, determining the operation type information corresponding to the text information according to the corresponding standard text;
[0214] According to the determined operation type information, the entity corresponding to the target object or the number of operations of the target object performing the specified operation is determined.
[0215] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, the method further includes:
[0216] Acquire a second sample image of a plurality of target object samples;
[0217] Performing a labeling process on each of the second sample images to label the text information to be recognized;
[0218] According to a preset second training method, training is performed based on the second sample image after annotation to obtain the text recognition model.
[0219] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, the method further includes:
[0220] Acquire a third sample image of a plurality of target object samples; wherein the third sample image includes a first layout of the target object samples;
[0221] Performing a labeling process on each of the third sample images to label the structured fields in the first layout;
[0222] According to a preset third training method, training processing is performed based on the third sample image after labeling processing to obtain the structured positioning model.
[0223] Optionally, when the computer executable instructions stored in the storage medium are executed by the processor, the method further includes:
[0224] Acquire a fourth sample image of a plurality of target object samples; wherein the fourth sample image includes a second layout of the target object;
[0225] Performing a labeling process on each of the fourth sample images to label a text area;
[0226] According to the preset fourth training method, training processing is performed based on the fourth sample image after annotation processing to obtain the text detection model.
[0227] When the computer executable instructions stored in the storage medium provided by one or more embodiments of the present application are executed by the processor, when an image to be identified including a target object with different layout formats is obtained, the layout information of the target object included in the image to be identified is determined, and the corresponding pre-trained processing model is used to perform recognition processing according to the determined layout information. Therefore, for target objects with different layout formats, by pre-training processing models for images with different layouts, when the image to be identified is obtained, different processing models can be used for targeted recognition processing according to the layout information of the target object included in the image to be identified, instead of using the same recognition method for all images; therefore, the accuracy of recognition is greatly improved, especially for images with structured layouts, the structured information can be accurately identified, which is conducive to improving the processing efficiency of the business and can better meet the current high-efficiency business processing needs.
[0228] It should be noted that the embodiments of the storage medium in this specification and the embodiments of the image recognition method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding image recognition method mentioned above, and the repeated parts will not be repeated.
[0229] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0230] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0231] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0233] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0234] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0235] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0236] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0237] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0238] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0239] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.
Claims
1. An image recognition method, characterized in that: include: Acquire an image of a target object to be identified; wherein the target object includes a first layout and a second layout with different layout formats; Determining layout information of the target object included in the image to be identified; If the layout information is the first layout, performing structured positioning processing on the image to be identified based on a pre-trained structured positioning model, and performing text recognition processing on the first target text to be identified obtained by the structured positioning processing to obtain first recognition result information; If the layout information is the second layout, performing text detection processing on the image to be recognized based on a pre-trained text detection model, and performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information; The performing structured positioning processing on the image to be identified based on the pre-trained structured positioning model includes: Inputting the image to be recognized into a pre-trained structured positioning model for structured positioning processing to obtain a positioning frame of the first target text to be recognized and field information of a structured field corresponding to the first target text; Performing image extraction processing according to the positioning frame to obtain a first sub-image including the first target text; The performing text recognition processing on the first target text to be recognized obtained by the structured positioning processing to obtain first recognition result information includes: Inputting the first sub-image and the corresponding field information into a pre-trained text recognition model for text recognition processing to obtain first recognition result information; Performing text detection processing on the image to be recognized based on a pre-trained text detection model includes: Inputting the image to be recognized into a pre-trained text detection model to perform text detection processing to obtain a text area; Performing image extraction processing according to the text area to obtain a second sub-image including the second target text to be recognized; The performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information includes: Inputting the second sub-image into a pre-trained text recognition model for text recognition processing to obtain second recognition result information; The method further comprises: Acquire a second sample image of a plurality of target object samples; Performing a labeling process on each of the second sample images to label the text information to be recognized; According to a preset second training method, training is performed based on the second sample image after the labeling process to obtain the text recognition model; Acquire a third sample image of a plurality of target object samples; wherein the third sample image includes a first layout of the target object samples; Performing a labeling process on each of the third sample images to label the structured fields in the first layout; According to a preset third training method, training processing is performed based on the third sample image after the annotation processing to obtain the structured positioning model; Acquire a fourth sample image of a plurality of target object samples; wherein the fourth sample image includes a second layout of the target object; Performing a labeling process on each of the fourth sample images to label a text area; According to the preset fourth training method, training processing is performed based on the fourth sample image after annotation processing to obtain the text detection model.
2. The method according to claim 1, characterized in that The step of determining the layout information of the target object included in the image to be identified includes: Classify the image to be identified according to a pre-trained classification model to obtain classification result information of the image to be identified; The layout information of the target object included in the image to be identified is determined according to the classification result information.
3. The method according to claim 2, characterized in that The method further comprises: Determine the position information of the target object in the image to be identified according to the classification result information; If it is determined according to the position information that the target object is not in a preset position in the image to be identified, correction processing is performed on the image to be identified.
4. The method according to claim 3, characterized in that The method further comprises: Acquire a first sample image of a plurality of target object samples; Performing a labeling process on each of the first sample images to label the layout information and orientation information of the target object sample in the first sample image; According to a preset first training method, training processing is performed based on the first sample image after labeling to obtain the classification model.
5. The method according to claim 1, characterized in that After performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information, the method further includes: The entity corresponding to the target object or the number of operations of the target object performing a designated operation is determined according to the second recognition result information.
6. The method according to claim 5, characterized in that Determining, according to the second recognition result information, the entity corresponding to the target object or the number of operations of the target object performing a specified operation, including: Determine the similarity between each text information in the second recognition result information and each standard text in the specified standard text library according to a preset method; If it is determined that the similarity is greater than a preset similarity threshold, determining the operation type information corresponding to the text information according to the corresponding standard text; According to the determined operation type information, the entity corresponding to the target object or the number of operations of the target object performing the specified operation is determined.
7. An image recognition device, characterized in that: include: An acquisition module, used for acquiring an image to be identified of a target object; wherein the target object includes a first layout and a second layout with different layout formats; A determination module, used for determining the layout information of the target object included in the image to be identified; A first recognition module is used for, if the layout information is the first layout, performing structured positioning processing on the image to be recognized based on a pre-trained structured positioning model, and performing text recognition processing on the first target text to be recognized obtained by the structured positioning processing to obtain first recognition result information; A second recognition module is used for, if the layout information is the second layout, performing text detection processing on the image to be recognized based on a pre-trained text detection model, and performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information; The first recognition module inputs the image to be recognized into a pre-trained structured positioning model for structured positioning processing, and obtains the positioning frame of the first target text to be recognized and the field information of the structured field corresponding to the first target text; Performing image extraction processing according to the positioning frame to obtain a first sub-image including the first target text; Inputting the first sub-image and the corresponding field information into a pre-trained text recognition model for text recognition processing to obtain first recognition result information; The second recognition module inputs the image to be recognized into a pre-trained text detection model to perform text detection processing to obtain a text area; Performing image extraction processing according to the text area to obtain a second sub-image including the second target text to be recognized; The performing text recognition processing on the second target text to be recognized obtained by the text detection processing to obtain second recognition result information includes: Inputting the second sub-image into a pre-trained text recognition model for text recognition processing to obtain second recognition result information; The device also includes: a second training module; The second training module obtains second sample images of multiple target object samples; Performing a labeling process on each of the second sample images to label the text information to be recognized; According to a preset second training method, training is performed based on the second sample image after the labeling process to obtain the text recognition model; The device further comprises: a third training module; The third training module obtains a third sample image of a plurality of target object samples; wherein the third sample image includes a first layout of the target object sample; Performing a labeling process on each of the third sample images to label the structured fields in the first layout; According to a preset third training method, training processing is performed based on the third sample image after the annotation processing to obtain the structured positioning model; The device further comprises: a fourth training module; The fourth training module acquires a fourth sample image of a plurality of target object samples; wherein the fourth sample image includes a second layout of the target object; Performing a labeling process on each of the fourth sample images to label a text area; According to the preset fourth training method, training processing is performed based on the fourth sample image after annotation processing to obtain the text detection model.
8. The device according to claim 7, characterized in that The determination module is specifically used for: Classify the image to be identified according to a pre-trained classification model to obtain classification result information of the image to be identified; The layout information of the target object included in the image to be identified is determined according to the classification result information.
9. The device according to claim 8, characterized in that The device also includes: a correction module; The correction module is used to determine the position information of the target object in the image to be identified according to the classification result information; If it is determined according to the position information that the target object is not in a preset position in the image to be identified, correction processing is performed on the image to be identified.
10. An image recognition device, characterized in that: include: a processor, and a memory arranged to store computer executable instructions; When the computer executable instructions are executed, the processor is caused to implement the steps of the method of any one of claims 1 to 6 above.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.
