An image processing method, apparatus, electronic device, and storage medium

Abstract

This application provides an image processing method, apparatus, electronic device, and storage medium. The method includes: obtaining text regions from an image to be detected, each text region including a rotation angle, a region width, and a region height; correcting the rotation angles of the text regions based on the rotation angles of multiple text regions and a preset first angle threshold to obtain a corrected rotation angle; and calibrating the orientation of the text regions based on the corrected rotation angle, region width, and region height to obtain a region to be detected. By obtaining text regions from the image to be detected and correcting the rotation angles of the text regions to ensure that the orientation of each text region in the image is consistent, and then calibrating the orientation of the text regions based on the corrected rotation angle, region width, and region height to obtain a region to be detected, the region to be detected is horizontal. The uniform and horizontal orientation of the region to be detected reduces errors and improves the accuracy and efficiency of detection.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of internet and computer technology, a vast amount of multimedia information appears in people's daily lives. Many businesses need to analyze and understand the information in images using computers to identify the relevant information. Current methods such as OCR (Optical Character Recognition) recognize text by directly recognizing the entire image. However, this often results in low image processing accuracy due to factors such as the orientation of the text in the recognition results. Summary of the Invention

[0003] The purpose of this invention is to provide an image processing method, apparatus, electronic device, and storage medium for directional calibration of text regions obtained from an image to obtain a region to be detected, thereby improving the accuracy of image processing.

[0004] In a first aspect, embodiments of this application provide an image processing method, comprising: obtaining text regions from an image to be detected, each text region including a rotation angle, a region width, and a region height; correcting the rotation angle of the text regions according to the rotation angles of multiple text regions and a preset first angle threshold to obtain a corrected rotation angle; and calibrating the orientation of the text regions according to the corrected rotation angle, region width, and region height to obtain the region to be detected.

[0005] In the above implementation process, text regions are obtained from the image to be detected, and the rotation angle of the text regions is corrected to ensure that the orientation of all text regions in the image to be detected is consistent. Then, based on the corrected rotation angle, region width, and region height, the orientation of the text regions is calibrated to obtain the region to be detected, ensuring that the region to be detected is horizontal. Because the region to be detected is consistent and horizontal, detection errors caused by inconsistent text orientation are avoided, thus improving the accuracy and efficiency of detection.

[0006] Optionally, in this embodiment of the application, the rotation angle of the text region is corrected according to the rotation angle of multiple text regions and a preset first angle threshold to obtain a corrected rotation angle, including: obtaining the difference between the maximum value and the minimum value of the rotation angle in multiple text regions according to the rotation angle of multiple text regions; if the difference is greater than the first angle threshold, the rotation angle of the text region whose rotation angle is less than the preset second angle threshold is corrected to obtain a corrected rotation angle.

[0007] In the above implementation process, the difference between the maximum and minimum rotation angles in multiple text regions is compared with a preset first angle threshold to correct the rotation angle of the text regions. Affine transformation is then performed based on the corrected rotation angle to unify the orientation of the corrected text regions, thereby improving the problem of low detection accuracy caused by inconsistent text orientations in the same image to be detected.

[0008] Optionally, in this embodiment of the application, the text region is oriented and calibrated according to the corrected rotation angle, width, and height of the text region to obtain the region to be detected, including: obtaining the aspect ratio of each text region according to the width and height of each text region; obtaining the average aspect ratio of multiple text regions according to the aspect ratio of each text region; and calibrating the text region according to the corrected rotation angle and the average aspect ratio to obtain the region to be detected.

[0009] In the above implementation process, the text region is oriented and calibrated according to the corrected rotation angle and the average aspect ratio of the text region, so that the regions to be detected are all horizontal text that can be directly input into the recognition network, thereby improving the image processing accuracy.

[0010] Optionally, in this embodiment, the text region is oriented and calibrated according to the corrected rotation angle and the average aspect ratio of the text region to obtain the region to be detected. This includes: performing an affine transformation on the text region according to the corrected rotation angle to obtain a text rectangle; if the average aspect ratio is greater than a preset aspect ratio threshold, rotating the text rectangle to obtain the region to be detected; if the average aspect ratio is less than the aspect ratio threshold, using the text rectangle as the region to be detected.

[0011] In the above implementation process, an affine transformation is performed on the text region based on the corrected rotation angle to obtain a text rectangle. The average aspect ratio of multiple text regions is then used to determine whether the text rectangle needs to be rotated. This ensures that the obtained detection regions are all horizontal text that can be directly input into the recognition network, improving image processing accuracy.

[0012] Optionally, in this embodiment of the application, after calibrating the orientation of the text region according to the corrected rotation angle, width and height of the text region to obtain the region to be detected, the method further includes: inputting the region to be detected into a preset orientation classification model to obtain the orientation classification result of each region to be detected; wherein, the orientation classification model is obtained by training a pre-collected set of orientation text samples; and performing text forward calibration on the region to be detected according to the orientation classification result.

[0013] In the above implementation process, the region to be detected is input into a preset orientation classification model to obtain the orientation classification result of each region to be detected. The orientation of the text in the region to be detected is classified by the binary orientation classification model. The text orientation is corrected according to the classification result so that the regions to be detected are all horizontal and positive text, thereby improving the subsequent recognition accuracy.

[0014] Optionally, in this embodiment of the application, after calibrating the orientation of the text region according to the corrected rotation angle, width, and height of the text region to obtain the region to be detected, the method further includes: obtaining the order product image information corresponding to the image to be detected; performing text recognition on the region to be detected to obtain the text recognition result; comparing the text recognition result with the order product image information to obtain a consistency detection result, wherein the consistency detection result characterizes whether the text recognition result in the image to be detected is consistent with the order product image information.

[0015] In the above implementation process, text recognition is performed on the detection area after orientation calibration and forward calibration to obtain text recognition results. By comparing the text recognition results with the corresponding order product image information, consistency detection results are obtained, which improves the verification efficiency. At the same time, it improves the problem of low consistency detection accuracy caused by incorrect orientation of the detection area, thus improving the accuracy of consistency detection.

[0016] Optionally, in this embodiment of the application, obtaining the text region from the image to be detected includes: obtaining the image to be detected, inputting the image to be detected into a preset instance segmentation model to obtain segmentation information; and segmenting the text region from the image to be detected based on the segmentation information.

[0017] In the above implementation process, the text region in the image to be detected is segmented by instance segmentation. This not only accurately segments the field information of different templates and different positions, but also adapts well to the situation where instances overlap.

[0018] Secondly, embodiments of this application also provide an image processing apparatus, including: an acquisition module, configured to acquire text regions from an image to be detected, each text region including a rotation angle, a region width, and a region height; an angle correction module, configured to correct the rotation angle of the text regions based on the rotation angles of multiple text regions and a preset first angle threshold, to obtain a corrected rotation angle; and a region processing module, configured to perform orientation calibration on the text regions based on the corrected rotation angle, region width, and region height, to obtain the region to be detected.

[0019] Thirdly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method described above.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the methods described above.

[0021] The image processing method, apparatus, electronic device, and storage medium provided in this application are used to obtain text regions from an image to be detected. The rotation angle of the text regions is corrected to ensure that the orientation of all text regions in the image to be detected is consistent. Then, based on the corrected rotation angle, region width, and region height, the orientation of the text regions is calibrated to obtain the detection region, ensuring that the detection region is horizontal. Because the detection region has a uniform and horizontal orientation, the problem of low detection accuracy caused by inconsistent text orientations within the same image to be detected is avoided, thus improving the accuracy and efficiency of detection. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the minimum outer rectangle of the text region provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram of the rotation angle of the text area provided in the embodiments of this application;

[0026] Figure 4 A schematic diagram of the process for correcting the rotation angle provided in an embodiment of this application;

[0027] Figure 5 A schematic diagram illustrating the text region orientation calibration process provided in this application embodiment;

[0028] Figure 6 A schematic diagram of the text forward calibration process provided in the embodiments of this application;

[0029] Figure 7A schematic diagram illustrating the process of acquiring an image to be detected, provided in an embodiment of this application;

[0030] Figure 8 A schematic diagram of the text recognition process provided in the embodiments of this application;

[0031] Figure 9 A schematic diagram of a segmented text region provided in an embodiment of this application;

[0032] Figure 10 This is a schematic diagram of the consistency information verification process provided in the embodiments of this application;

[0033] Figure 11 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0034] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0037] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0038] Please see Figure 1 The illustration shows a flowchart of an image processing method provided in an embodiment of this application. The image processing method provided in this application can be applied to an electronic device, which may include a terminal and a server; wherein the terminal may specifically be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; the server may specifically be an application server or a web server. The image processing method may include the following steps:

[0039] Step S110: Obtain text regions from the image to be detected. Each text region includes a rotation angle, region width, and region height.

[0040] The image to be inspected can be an image acquired by a data acquisition device from which valid information needs to be extracted. The valid information in the image can be used for business processing such as consistency verification. The data acquisition device can be a network camera (IP CAMERA, IPC) or a handheld operating device such as a mobile phone.

[0041] Text regions are obtained from an image to be detected using image segmentation and matting tools. An image to be detected may include multiple text regions, which can be specifically an image block within the image. Each text region represents a set of text data on the image to be detected. Each text region includes attribute information such as rotation angle, region width, and region height.

[0042] Step S120: Based on the rotation angles of multiple text regions and a preset first angle threshold, the rotation angle of the text region is corrected to obtain the corrected rotation angle.

[0043] After obtaining the text region, an affine transformation needs to be performed on it according to the rotation angle to obtain a rectangular image patch, i.e., the region to be detected. During the affine transformation, it's impossible to determine whether the text in the region is skewed to the left or right. Therefore, after rotating the text region according to the rotation angle, the resulting rectangular region could contain either horizontal or vertical text. Thus, the rotation angle needs to be adjusted to obtain a corrected rotation angle, ensuring that after rotating the text region according to the corrected rotation angle, the orientation of all text regions in the image to be detected is consistent.

[0044] For example, adjusting the rotation angle involves obtaining the rotation angles of multiple text regions in the image to be detected. If the difference between the maximum and minimum rotation angles in a text region exceeds a preset first angle threshold, the rotation angle of the text region needs to be corrected to obtain a corrected rotation angle. If the difference between the maximum and minimum rotation angles in a text region does not exceed the preset first angle threshold, it indicates that the deflection direction of all text regions is consistent, and no correction of the rotation angle is required. This corrected rotation angle can then be used for subsequent steps.

[0045] Step S130: Based on the corrected rotation angle, width, and height of the text region, perform orientation calibration on the text region to obtain the region to be detected.

[0046] After obtaining the corrected rotation angle, the text regions are rotated according to this angle, ensuring that the orientation of all text regions in the image to be detected is consistent. However, some text regions may uniformly display horizontal text or vertical text. In actual business processing, horizontal text is usually processed, as vertical text may cause inaccurate detection. Therefore, it is necessary to perform orientation calibration on the text regions to ensure they are horizontal text.

[0047] For example, based on the width and height of each text region, the average aspect ratio of the text region in the image to be detected is obtained. The orientation of the text region is then calibrated based on the corrected rotation angle and the average aspect ratio to obtain the region to be detected.

[0048] In the above implementation process, text regions are obtained from the image to be detected, and the rotation angle of the text regions is corrected to ensure that the orientation of each text region in the image to be detected is consistent. Then, based on the corrected rotation angle, region width, and region height, the orientation of the text regions is calibrated to obtain the region to be detected, ensuring that the region to be detected is horizontal. Because the region to be detected is uniformly oriented and horizontal, detection errors caused by inconsistent text orientation are avoided, thus improving the accuracy and efficiency of detection.

[0049] Optionally, in this embodiment of the application, the rotation angle of the text region is corrected according to the rotation angle of multiple text regions and a preset first angle threshold to obtain a corrected rotation angle, including: obtaining the difference between the maximum value and the minimum value of the rotation angle in multiple text regions according to the rotation angle of multiple text regions; if the difference is greater than the first angle threshold, the rotation angle of the text region whose rotation angle is less than the preset second angle threshold is corrected to obtain a corrected rotation angle.

[0050] Please see Figure 2 The diagram shown is a schematic diagram of the minimum outer rectangle of the text area provided in the embodiment of this application.

[0051] In the specific implementation process: the `cv2.minAreaRect` function from the OpenCV computer vision and machine learning software library is used to obtain the minimum bounding rectangle of the text region. Specifically, based on the edges or contours of the image blocks corresponding to the text region, the minimum bounding rectangle of each text region is extracted. If a text region has one bounding rectangle, that bounding rectangle is the minimum bounding rectangle; if a text region has multiple bounding rectangles, the smallest bounding rectangle among these is taken as the minimum bounding rectangle of the object. The minimum bounding rectangle information includes, for example, `(center(x,y),(width,height),θ)`; where `center(x,y)` is the coordinates of the rectangle's center point; `(width,height)` are the rectangle's width and height; and `θ` is the rectangle's rotation angle. Note that `width` and `height` are not calculated based on length, but rather on the first edge encountered by `θ`. The four vertices of the rectangle are calculated using the `boxPoints` tool, or mathematically based on the bounding rectangle information, such as the center point coordinates, width, and height.

[0052] Please see Figure 3 The diagram shows the rotation angle of the text area provided in the embodiment of this application.

[0053] like Figure 3 As shown, each text region in the image to be detected has a corresponding rotation angle. There are two text regions with a rotation angle of -30 degrees and two text regions with a rotation angle of -60 degrees. It is impossible to determine whether the text direction of the text region is to the left or right. However, the affine transformation is performed in the same direction. If the text directions of text regions with the same rotation angle are inconsistent, it will lead to the mixing of horizontal and vertical text in the text region after the affine transformation. Therefore, it is necessary to correct the rotation angle.

[0054] Please see Figure 4 The diagram shown is a flowchart illustrating the process of correcting the rotation angle provided in an embodiment of this application.

[0055] Obtain the minimum bounding rectangle of the text region in the text to be detected, and the rotation angle θ corresponding to the minimum bounding rectangle. Obtain the difference between the maximum rotation angle max_θ and the minimum rotation angle min_θ in multiple text regions, and label the difference as diff_θ.

[0056] The algorithm determines whether the difference value diff_θ is greater than a first angle threshold. If the difference is greater than the first angle threshold, the rotation angle of all text regions whose rotation angle is less than a preset second angle threshold is corrected to obtain a corrected rotation angle. Specific correction methods include increasing the rotation angle of the text region by 90 degrees. For example, if the rotation angle is -80 degrees, the second angle threshold can be -45 degrees. If the rotation angle is less than the preset second angle threshold, the rotation angle is increased by 90 degrees, resulting in a corrected rotation angle of 10 degrees.

[0057] If the difference is less than the first angle threshold, it indicates that the text direction of the text region in the image to be detected is uniform, and there is no need to correct the rotation angle of the text region. The original rotation angle is used as the correction rotation angle. Finally, an affine transformation is performed using the correction rotation angle to obtain the text instance, i.e., the text rectangle.

[0058] The first angle threshold can be set according to actual needs, such as 60 degrees, 65 degrees or 70 degrees.

[0059] In the above implementation process, the difference between the maximum and minimum rotation angles in multiple text regions is compared with a preset first angle threshold to correct the rotation angle of the text regions. Affine transformation is then performed based on the corrected rotation angle to ensure that the orientation of the corrected text regions is consistent, thus avoiding the problem of low detection accuracy caused by inconsistent text orientations in the same image to be detected.

[0060] Optionally, in this embodiment of the application, the text region is oriented and calibrated according to the corrected rotation angle, width, and height of the text region to obtain the region to be detected, including: obtaining the aspect ratio of each text region according to the width and height of each text region; obtaining the average aspect ratio of multiple text regions according to the aspect ratio of each text region; and calibrating the text region according to the corrected rotation angle and the average aspect ratio to obtain the region to be detected.

[0061] Please see Figure 5 The illustration shows a flowchart of text region orientation calibration provided in an embodiment of this application.

[0062] In the specific implementation process: After unifying the orientation of the corrected text regions, the text regions may be unified as horizontal text or vertical text. However, the rectangular regions input to the recognition network should be horizontal text, so the text regions are first converted into horizontal text according to the aspect ratio.

[0063] Specifically, for example, the width *w* and height *h* of each text region are collected, and the aspect ratio *w / h* of each text region is calculated and stored. The sum of the aspect ratios of multiple text regions is divided by the number of text regions *n* to obtain the average aspect ratio *(w / h) / n*. The average aspect ratio can characterize the text orientation trend of multiple text regions in the image to be detected. Based on the corrected rotation angle and the average aspect ratio of the text regions, the orientation of the text regions is calibrated to obtain the region to be detected.

[0064] Optionally, in this embodiment, the text region is oriented and calibrated according to the corrected rotation angle and the average aspect ratio of the text region to obtain the region to be detected. This includes: performing an affine transformation on the text region according to the corrected rotation angle to obtain a text rectangle; if the average aspect ratio is greater than a preset aspect ratio threshold, rotating the text rectangle to obtain the region to be detected; if the average aspect ratio is less than the aspect ratio threshold, using the text rectangle as the region to be detected.

[0065] In the specific implementation process: An affine transformation is performed on the text region based on the corrected rotation angle to obtain a text rectangle. The text rectangle is a text region with a uniform orientation. If the average aspect ratio is greater than a preset aspect ratio threshold, the text rectangle is rotated to obtain the region to be detected. The aspect ratio threshold can be set according to actual needs, for example, to 1, or it can be 0.9 or 1.1, etc. This application uses an aspect ratio threshold of 1 as an example. If the average aspect ratio (w / h) / n is greater than 1, it indicates that most text regions are vertical text. The text regions are then rotated uniformly by 90 degrees clockwise or counterclockwise to make all text regions horizontal text, thus obtaining the region to be detected.

[0066] If the average aspect ratio (w / h) / n is not greater than the preset aspect ratio threshold of 1, it indicates that the text area is all horizontal text and there is no need to select the text area.

[0067] In the above implementation process, an affine transformation is performed on the text region based on the corrected rotation angle to obtain a text rectangle. The average aspect ratio of multiple text regions is then used to determine whether the text rectangle needs to be rotated. This ensures that the obtained detection regions are all horizontal text that can be directly input into the recognition network, improving image processing accuracy.

[0068] Optionally, in this embodiment of the application, after calibrating the orientation of the text region according to the corrected rotation angle, width and height of the text region to obtain the region to be detected, the method further includes: inputting the region to be detected into a preset orientation classification model to obtain the orientation classification result of each region to be detected; wherein, the orientation classification model is obtained by training a pre-collected set of orientation text samples; and performing text forward calibration on the region to be detected according to the orientation classification result.

[0069] In the specific implementation process: when the text area is vertical text, rotate the text area 90 degrees clockwise or counterclockwise to make the text area all horizontal text. At this time, the horizontal text may be upright text, that is, text with 0 degrees in the horizontal direction, or it may be upside down text, that is, text with 180 degrees in the horizontal direction. In this case, the upside down text needs to be calibrated to make the text in the area to be detected upright text.

[0070] The region to be detected is input into a pre-defined orientation classification model to obtain the orientation classification result for each region. The orientation classification result indicates whether the region to be detected is upright or inverted text. The orientation classification model is obtained by training a pre-collected set of orientation text samples; the orientation text sample set includes upright text samples and / or inverted text samples.

[0071] Based on the orientation classification results, the text to be detected is calibrated to be positive. For example, if the orientation classification results indicate that the text in the area to be detected is positive, then there is no need to calibrate the text in the area to be detected; if the orientation classification results indicate that the text in the area to be detected is inverted, then the area to be detected is rotated 180 degrees to make the area to be detected be positive.

[0072] Please see Figure 6 The illustration shows a schematic diagram of the text forward calibration process provided in an embodiment of this application.

[0073] In an optional embodiment, for detection regions whose smallest bounding rectangle is close to a square, the orientation classification model may be inaccurate. Therefore, this solution selects the N detection regions with the largest aspect ratio for orientation classification detection, avoiding the use of detection regions that are close to a square for orientation classification detection, thus improving the accuracy of orientation classification and avoiding the huge workload of classifying all detection regions.

[0074] The following is an example: the aspect ratio of each region to be detected is obtained, and a preset number of regions to be detected with the largest aspect ratio is determined; the preset number of regions to be detected are input into a preset orientation classification model to obtain the orientation classification result of each region to be detected; and the text is positively calibrated according to the orientation classification result.

[0075] If the number of regions to be detected is less than the preset number, then all regions to be detected will be input into the preset directional classification model.

[0076] The orientation classification model can be a binary classification model. This model uses a binary classification to vote on the text orientation of the regions to be detected, and then corrects the text orientation based on the voting results. The preset number can be set according to actual needs; for example, the five regions with the largest aspect ratios can be used for orientation classification detection. It can also be set to six or eight, etc., and this application does not impose any restrictions on this.

[0077] In the above implementation process, the region to be detected is input into a preset orientation classification model to obtain the orientation classification result of each region to be detected. The orientation of the text in the region to be detected is classified by the binary orientation classification model. The text orientation is corrected according to the classification result so that the regions to be detected are all horizontal and positive text, thereby improving the subsequent recognition accuracy.

[0078] Optionally, in this embodiment of the application, after calibrating the orientation of the text region according to the corrected rotation angle, width, and height of the text region to obtain the region to be detected, the method further includes: obtaining the order product image information corresponding to the image to be detected; performing text recognition on the region to be detected to obtain the text recognition result; comparing the text recognition result with the order product image information to obtain a consistency detection result, wherein the consistency detection result characterizes whether the text recognition result in the image to be detected is consistent with the order product image information.

[0079] In the specific implementation process: The image processing method of this application can be applied to the consistency information recognition scenario. For example, when a user places an order for a certain product and obtains the image information of the ordered product, the on-site operator needs to compare the physical product information with the image information of the ordered product in order to accurately deliver the ordered product to the user.

[0080] You can obtain the order product image information corresponding to the image to be detected, or you can first obtain the order product image information, and then obtain the image to be detected corresponding to the order product image information for comparison.

[0081] Please see Figure 7 The illustration shows a flowchart of the process for acquiring an image to be detected, as provided in an embodiment of this application.

[0082] Among them, "collecting the unique code of the product" refers to scanning the barcode or QR code on the product packaging to obtain the unique code of the product on the current platform. Each product has a unique barcode when it is circulating in the warehouse, which is associated with the product information of the user's order.

[0083] "Collecting products" refers to collecting product label images, product appearance images, and product packaging images.

[0084] Please see Figure 8 The illustration shows a flowchart of text recognition provided in an embodiment of this application.

[0085] Before comparing the area to be detected with the image information of the ordered product, the area to be detected needs to be recognized as text. The recognition process includes: first, using an image preprocessing model to eliminate the influence of color channels and convert the image to grayscale; and second, due to the limitations of RNN, the image height needs to be fixed, so the height of the area to be detected is also set.

[0086] Image features are extracted using a CNN feature extraction module, where a lightweight model, Mobilenet, is used as the backbone to learn image features.

[0087] The text in the image has a sequential relationship, and phrases, sentences, or sentences are related by time sequence. The relationship is learned through the RNN character sequence representation module.

[0088] The CTCLoss module selects the optimal path to determine the classification result. For example, each character is segmented and then fed into a fully connected layer for character classification. Since the spacing between characters in the image is not fixed, there may be cases where spaces are cut out or a character is cut in half. Therefore, CTCLoss is used to solve this misalignment segmentation problem.

[0089] The character sequence post-processing module obtains the local optimal classification result for the region to be detected. When obtaining the text recognition result, it is necessary to merge duplicate characters and remove whitespace characters. For example, if the text recognition result is a_p_p_l_e, removing the whitespace character _ will result in the final text recognition result being apple; if the prediction is a_pp_p_l_e, merging the duplicate character 'p' and removing the whitespace character _ will result in the final text recognition result being apple; if the prediction is a_pp_l_e, the final text recognition result being aple is an incorrect result. In this case, further fine-tuning of the model parameters is needed to find a reasonable optimal classification result.

[0090] The text recognition results are compared one by one with the strings or numbers in the order product image information to obtain the consistency detection results. The consistency detection results represent whether the text recognition results in the image to be detected are consistent with the order product image information. The consistency detection results can also be the degree of consistency between the text recognition results in the image to be detected and the order product image information, such as a score or probability, and then determine whether they are consistent based on a preset threshold.

[0091] In an optional embodiment, when the consistency detection result indicates that the text recognition result in the image to be detected is inconsistent with the image information of the ordered product, a prompt message or alarm can be issued to remind the operator.

[0092] In the above implementation process, text recognition is performed on the detection area after orientation calibration and forward calibration to obtain text recognition results. By comparing the text recognition results with the corresponding order product image information, consistency detection results are obtained, which improves the verification efficiency. At the same time, it improves the problem of low consistency detection accuracy caused by incorrect orientation of the detection area, thus improving the accuracy of consistency detection.

[0093] Optionally, in this embodiment of the application, obtaining the text region from the image to be detected includes: obtaining the image to be detected, inputting the image to be detected into a preset instance segmentation model to obtain segmentation information; and segmenting the text region from the image to be detected based on the segmentation information.

[0094] Please see Figure 9 The diagram shown illustrates a segmented text region provided in an embodiment of this application.

[0095] In the specific implementation process: the image preprocessing module outputs an input that meets the model input requirements. For example, the maximum length and width of the image are constrained by the sampling algorithm, and then the images are stitched together into a 4-dimensional tensor (dimension) as the input of the model. For example, the longest side of the input is set to 1333, and the tensor is set to 1*3*1333*800.

[0096] The instance segmentation module inputs the preprocessed image into a preset instance segmentation model for segmentation to obtain the coordinate data of each text, i.e., the segmentation information, such as the coordinates of the polygon enclosing the text.

[0097] The instance matting module segments the image to be detected based on the coordinate data of each text element to obtain the text region.

[0098] Understandably, obtaining text regions from an image to be detected can also utilize models such as FCENet or DBNet++, semantic segmentation algorithms such as FCN or Knet, and instance segmentation methods such as Mask-RCNN. However, traditional text detection models output text boxes as single-class targets, requiring complex logical rules and resulting in poor accuracy when applied to fields such as product numbers and sizes. Semantic segmentation methods also involve complex post-processing. Therefore, in the above implementation, instance segmentation is used to segment the text regions in the image to be detected. This not only accurately segments field information from different templates and locations but also better adapts to situations where instances overlap.

[0099] Please see Figure 10 The diagram shown illustrates the consistency information verification process provided in the embodiments of this application.

[0100] In an optional embodiment, uniquely identified label images are selected from the acquired images. A pre-trained text detection model is used to perform structured detection and correction of consistency information. A pre-trained text orientation model is then used to determine the text orientation of the consistency information, ensuring that text regions containing consistency information are all horizontal and oriented text. Finally, a pre-trained character recognition model is used to perform character recognition on the consistency information that has undergone orientation determination, obtaining the character recognition result.

[0101] The system retrieves the user's order information from the storage unit, verifies the consistency between the text recognition result and the order information, obtains the consistency detection result, and completes the consistency detection of the user's order.

[0102] Please see Figure 11 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this application; this application provides an image processing apparatus 200, including:

[0103] The acquisition module 210 is used to obtain text regions from the image to be detected, each text region including a rotation angle, region width and region height;

[0104] The angle correction module 220 is used to correct the rotation angle of the text region based on the rotation angle of multiple text regions and a preset first angle threshold, so as to obtain the corrected rotation angle.

[0105] The region processing module 230 is used to perform orientation calibration on the text region based on the corrected rotation angle, region width and region height of the text region to obtain the region to be detected.

[0106] Optionally, in this embodiment of the application, the image processing device, the angle correction module 220, is specifically used to obtain the difference between the maximum and minimum rotation angles of multiple text regions based on the rotation angles of multiple text regions; if the difference is greater than a first angle threshold, the rotation angle of the text regions whose rotation angle is less than a preset second angle threshold is corrected to obtain the corrected rotation angle.

[0107] Optionally, in this embodiment of the application, the image processing device, the region processing module 230, is specifically used to obtain the aspect ratio of each text region based on the region width and region height of each text region; obtain the average aspect ratio of multiple text regions based on the aspect ratio of each text region; and perform orientation calibration on the text regions based on the corrected rotation angle and the average aspect ratio of the text regions to obtain the region to be detected.

[0108] Optionally, in this embodiment of the application, the image processing device, the region processing module 230, is further configured to perform an affine transformation on the text region according to the corrected rotation angle to obtain a text rectangle; if the average aspect ratio is greater than a preset aspect ratio threshold, the text rectangle is rotated to obtain a region to be detected; if the average aspect ratio is less than the aspect ratio threshold, the text rectangle is used as a region to be detected.

[0109] Optionally, in this embodiment of the application, the image processing device further includes: a text forward calibration module, used to input the region to be detected into a preset direction classification model to obtain the direction classification result of each region to be detected; wherein, the direction classification model is obtained by training a pre-collected set of direction text samples; and the region to be detected is subjected to text forward calibration according to the direction classification result.

[0110] Optionally, in this embodiment of the application, the image processing device further includes a consistency detection module, used to obtain the order product image information corresponding to the image to be detected; perform text recognition on the area to be detected to obtain the text recognition result; compare the text recognition result with the order product image information to obtain a consistency detection result, wherein the consistency detection result characterizes whether the text recognition result in the image to be detected is consistent with the order product image information.

[0111] Optionally, in this embodiment of the application, the image processing device, the acquisition module 210, is specifically used to acquire the image to be detected, input the image to be detected into a preset instance segmentation model to obtain segmentation information, and segment the text region from the image to be detected based on the segmentation information.

[0112] It should be understood that this device corresponds to the image processing method embodiments described above and is capable of performing the various steps involved in the above method embodiments. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0113] Please see Figure 12 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0114] This application also provides a storage medium storing a computer program, which is executed by a processor to perform the above-described method.

[0115] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0116] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, given the several embodiments provided in this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0117] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0118] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Text regions are obtained from the image to be detected, each text region including a rotation angle, region width, and region height; Based on the rotation angles of multiple text regions and a preset first angle threshold, the rotation angle of the text regions is corrected to obtain a corrected rotation angle; Based on the corrected rotation angle, width, and height of the text region, the text region is oriented and calibrated to obtain the region to be detected. Based on the rotation angles of multiple text regions and a preset first angle threshold, the rotation angles of the text regions are corrected to obtain a corrected rotation angle, including: Based on the rotation angles of the multiple text regions, obtain the difference between the maximum and minimum rotation angles of the multiple text regions; If the difference is greater than the first angle threshold, then the rotation angle of the text region whose rotation angle is less than the preset second angle threshold is corrected to obtain the corrected rotation angle; Based on the corrected rotation angle, width, and height of the text region, the text region is oriented and calibrated to obtain the region to be detected, including: The aspect ratio of each text region is obtained based on its width and height. Based on the aspect ratio of each text region, the average aspect ratio of multiple text regions is obtained; Based on the corrected rotation angle of the text region and the average aspect ratio, the text region is oriented and calibrated to obtain the region to be detected.

2. The method according to claim 1, characterized in that, Based on the corrected rotation angle of the text region and the average aspect ratio, the text region is oriented and calibrated to obtain the region to be detected, including: Affine transformation is performed on the text region according to the corrected rotation angle to obtain a text rectangle; If the average aspect ratio is greater than a preset aspect ratio threshold, the text rectangle is rotated to obtain the region to be detected. If the average aspect ratio is less than the aspect ratio threshold, then the text rectangle is taken as the region to be detected.

3. The method according to claim 1, characterized in that, After calibrating the orientation of the text region according to the corrected rotation angle, width, and height of the text region to obtain the region to be detected, the method further includes: The region to be detected is input into a preset direction classification model to obtain the direction classification result for each region to be detected; wherein, the direction classification model is obtained by training a pre-collected set of direction text samples; Based on the directional classification results, the text in the region to be detected is calibrated in a positive direction.

4. The method according to claim 1, characterized in that, After calibrating the orientation of the text region according to the corrected rotation angle, width, and height of the text region to obtain the region to be detected, the method further includes: Obtain the order product image information corresponding to the image to be detected; Text recognition is performed on the area to be detected to obtain the text recognition result; The text recognition result is compared with the image information of the ordered product to obtain a consistency detection result. The consistency detection result indicates whether the text recognition result in the image to be detected is consistent with the image information of the ordered product.

5. The method according to claim 1, characterized in that, The process of obtaining the text region from the image to be detected includes: Obtain the image to be detected, input the image to be detected into a preset instance segmentation model, and obtain segmentation information; Based on the segmentation information, the text region is segmented from the image to be detected.

6. An image processing apparatus, characterized in that, include: An acquisition module is used to obtain text regions from the image to be detected, each text region including a rotation angle, region width and region height; The angle correction module is used to correct the rotation angle of the text region based on the rotation angles of multiple text regions and a preset first angle threshold, so as to obtain the corrected rotation angle. The region processing module is used to perform orientation calibration on the text region based on the corrected rotation angle, region width, and region height to obtain the region to be detected. The correction angle module is specifically used to obtain the difference between the maximum and minimum rotation angle values ​​in the multiple text regions based on their rotation angles. If the difference is greater than the first angle threshold, then the rotation angle of the text region whose rotation angle is less than the preset second angle threshold is corrected to obtain the corrected rotation angle; The region processing module is specifically used to obtain the aspect ratio of each text region based on its width and height; to obtain the average aspect ratio of multiple text regions based on the aspect ratio of each text region; and to perform orientation calibration on the text regions based on the corrected rotation angle of the text regions and the average aspect ratio to obtain the region to be detected.

7. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 5.

Images