Express bill visual detection method based on direction perception

Abstract

The invention relates to an express waybill visual detection method based on direction perception. The method comprises the following four steps: S1, constructing an express waybill image data set; s2, building an express bill detection model based on direction perception, and replacing an original horizontal frame detection head with a rotating frame detection head and a key point detection head based on improvement of a horizontal frame target detection model; s3, training the established express bill detection model based on direction perception until the model converges; s4, reasoning an image containing the express waybill by using the trained express waybill detection model to obtain the position and key point information of the express waybill, regarding the vertex closest to the key point in the vertexes of the rotating frame as the corresponding point of the vertex at the upper left corner of the upright express waybill, forming four groups of corresponding points clockwise by all the vertexes, establishing an affine transformation matrix, and calculating the position of the express waybill according to the affine transformation matrix. And obtaining a forward express bill image. The method has the advantages that accurate positioning of the position and the direction of the express waybill can be efficiently realized without designing a direction classifier, so that the forward express waybill image is directly obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more specifically, to a courier bill visual detection method based on direction perception. Background Art

[0002] With the rapid development of the logistics industry, the sorting and distribution work of express delivery points has become increasingly arduous. Manual sorting is inefficient and costly. Automated sorting is also difficult to accurately locate the position and direction of express orders using ordinary target detection technology due to the messy placement of express parcels. Most of the existing visual detection methods for express orders use traditional image processing or deep learning-based target detection technology. Traditional image processing methods generally separate the express order from the background through binarization, and combine morphological processing such as corrosion and expansion to make the outline of the express order clearer. Finally, the maximum stable extreme region (MEST) and stroke width change (SWT) are combined for detection. This type of method has two fatal problems in actual implementation: one is that the image processing algorithm parameters are selected based on experience and have poor generalization ability; the other is that the algorithm has low operating efficiency and it is difficult to ensure real-time performance. In terms of deep learning, Wang Jianyu used YOLOV5 to realize express order detection in the article "Research and Application of Text Detection and Recognition Technology Based on Deep Learning". This method uses a horizontal bounding box, which will result in a large amount of redundant background in the express order detection results, and the direction of the express order cannot be directly determined. The express order needs to be corrected through post-processing, which is relatively cumbersome. The patented method with application number CN202310783741 uses a rotating target detection method for sorting. Although it reduces the background redundancy of the courier bill detection results, it still cannot directly determine the direction of the courier bill. The courier bill is often reversed after the rotation boundary box is turned upright, which is not conducive to the subsequent OCR recognition of the courier bill. The existing method to solve this reversed problem usually uses a text direction classifier in the OCR process, which undoubtedly increases the computational burden of the courier bill OCR and increases the total time of courier bill sorting. Summary of the invention

[0003] In order to overcome the limitations of the prior art, the present invention provides a courier visual detection method based on direction perception, which can directly obtain a courier image with a positive text direction. The method mainly includes: S1, constructing a courier image dataset; S2, building a courier detection model based on direction perception, which is improved based on the existing conventional target detection model, using a rotating box detection head and a key point detection head to replace the original horizontal box detection head, and changing the original loss function horizontal box loss to a rotating box loss, and adding a key point detection loss; S3, training the constructed courier detection model based on direction perception; S4, using the trained courier detection model to infer the image containing the courier, obtain the courier position and key point information, and regard the vertex closest to the key point in the rotating box vertex as the corresponding point of the upper left corner vertex of the upright courier, all vertices form 4 groups of corresponding points in a clockwise direction, establish an affine transformation matrix, and obtain the positive courier image.

[0004] Further, the step S1 specifically includes: S11, collecting express delivery images and integrating them into an initial data set, including express delivery images taken in different environments and at different resolutions; S12, expanding the initial data set and annotating it, using the circumscribed rectangular box of the quadrilateral annotated with a polygon annotation tool as the target box for rotation target detection and regression, and using the upper left corner vertex of the express delivery order annotated with a point annotation tool as the key point.

[0005] Furthermore, in the step S2, a courier detection model based on direction perception is built, which can improve the existing common horizontal box-based target detection models, such as YOLO, SSD, etc. Their network structures usually include three parts: backbone, neck and head. The head part, i.e., the detection head, can be improved. For each original horizontal box detection head at different scales, a parallel rotating box detection head and a key point detection head are used to replace them. In addition to predicting the position of the bounding box and the target category, the rotating box detection head also predicts the rotation angle of the bounding box. The key point detection head is used to locate the upper left corner vertex corresponding to the courier when the courier is upright.

[0006] Further, the step S3 specifically includes: S31, taking a training image from the express order image data set each time to the express order detection model based on direction perception, inferring to obtain the prediction result of the express order, calculating the loss between the model prediction result and the sample annotation information, and the loss function of the express order visual detection model based on direction perception is composed of the classification loss of target detection, the rotation box regression loss, the key point regression loss and the confidence loss; S32, using the Adam optimizer, the initial learning is 0.01, and the learning rate is adjusted by using the learning rate preheating and cosine annealing strategy, and the weights of the express order detection model based on direction perception are updated by back propagation; S33, repeating the above steps until the model converges to obtain a trained express order detection model.

[0007] Further, in the step S4, the specific steps include: S41, using the trained express delivery order detection model to infer the image containing the express delivery order, obtain the express delivery order position and key point information, and convert the coordinate representation of the express delivery order position from the coordinates of the center point, width, height and rotation angle to the pixel coordinates of the four vertices of the rotating box; S42, calculating the distance between the four coordinate points and the key points of the rotated rectangular box detected by the express delivery order, and regard the vertex with the smallest distance as the corresponding point of the upper left corner vertex of the upright express delivery order. All vertices form four groups of corresponding points in a clockwise direction, and establish an affine transformation matrix. According to the affine transformation, the forward express delivery order image can be obtained.

[0008] Beneficial effects of the present invention:

[0009] 1. The present invention proposes a general method that can be used to improve any existing horizontal box-based object detection network, replacing the horizontal box detection head with a rotating box detection head and a key point detection head to accurately locate the position and direction of the express delivery order.

[0010] 2. The present invention uses a rotating frame to locate the position of the express delivery form, which has higher robustness than traditional image processing methods. Compared with the target detection method using a horizontal frame, it reduces redundant background and can achieve more accurate positioning when the position of the express delivery form is severely tilted.

[0011] 3. The present invention determines the direction of the express order through the position information of the rotation box and the key points, and establishes an affine transformation to directly obtain the positive express order. There is no need to use a direction classifier or other methods to determine the direction of the express order, which is more convenient and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 A flow chart of the main steps of the method provided in this application;

[0014] Figure 2 This is a schematic diagram of the structure of the courier order detection model based on direction perception used in this application;

[0015] Figure 3 This is the result image of the express delivery order detected by the existing detection method through the horizontal box and the rotation box;

[0016] Figure 4This is an example of a courier order with the same angle but reversed in the detection result of the existing rotation target detection model;

[0017] Figure 5 This is the inference result graph obtained by the direction-aware express delivery order detection model used in this application and the upright image obtained through affine transformation. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0019] In the following introduction, the terms "first" and "second" are only used for descriptive purposes and are not to be understood as indicating or implying relative importance. The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes may be made to the functions and arrangements of the elements described without departing from the scope of the present application. Various processes or components may be appropriately omitted, substituted or added to each example. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted or combined. In addition, the features described in some examples may be combined in other examples.

[0020] The embodiment of the present application provides a method for visual detection of express delivery orders based on direction perception. The present invention can be improved based on the existing common target detection model based on the horizontal frame. Their network structure usually includes three parts: backbone, neck and head. The improved head part is the detection head. In this implementation case, YOLOV8 is used as the basic model for modification, but the present invention is not limited to YOLOV8, and can also be modified on the basis of other target detection models, such as YOLOV5, YOLOV7, SSD, etc. The present invention can be applied to an image processing device, which can be an independent device, or can be integrated into an electronic terminal or other device with image processing capabilities. Optionally, the operating system carried by the device or electronic terminal integrating the image processing device can be a windows system, an android system or other operating system, which is not limited here.

[0021] Figure 1 FIG. 1 is a flow chart of the main steps of a method for visually detecting an express delivery form based on direction perception according to an embodiment of the present invention. Figure 1 , the method comprising:

[0022] S1. Build a courier image dataset;

[0023] S2. Build a direction-aware express delivery order detection model, use YOLOV8s as the basic model, and use the rotating box detection head and key point detection head to replace the original horizontal box target detection head;

[0024] S3, training a courier order detection model based on direction perception;

[0025] S4. Use the trained express delivery order detection model to infer the image containing the express delivery order, obtain the position and key point information of the express delivery order, regard the vertex closest to the key point in the rotation box as the corresponding point of the upper left corner vertex of the upright express delivery order, form 4 groups of corresponding points in clockwise direction, establish an affine transformation matrix, and obtain the forward express delivery order image.

[0026] The step S1 specifically includes:

[0027] S11. Collect the data set of express delivery orders. The same express delivery order is photographed in different environments and with different resolutions to expand the data set. There are two ways to detect express delivery orders: horizontal frame and rotation frame. These two detection methods correspond to different data annotation methods. Figure 3 As shown in the figure, it can be seen from the results of the rotation box and horizontal box detection that the horizontal box detection is prone to contain a lot of background, which requires further processing and positioning, which is cumbersome. In comparison, the rotation box can locate the express order more accurately. Therefore, the express order is annotated with the polygon tool of Labelme, and the quadrilateral box is used as the regression box for the rotation target detection. In order to solve the problem of uncertain text direction in the rotation box detection, a key point is marked in the upper left corner of the positive direction of each express order to determine the text direction.

[0028] S12. Use OpenCV to read the image and its size, and divide the quadrilateral coordinates and key point coordinates in the Json tag file by the width and height of the image for normalization. According to the format required by YOLOv8, save the category index, normalized quadrilateral coordinates, key point coordinates and their visibility information into a TXT file with the same name as the image. Finally, divide the 1,335 express delivery images into a training set and a validation set in a ratio of 9:1.

[0029] The step S2 specifically includes: Figure 2 The figure shows the model structure used in the present invention. The present invention selects YOLOV8s as the backbone network of the present invention. YOLOV8 uses PAFPN as Neck. The model used in the present invention replaces the original horizontal target detection box detection head with the rotation box detection head and the key point detection head. The key point detection head is used to achieve the regression of the key point. Each key point branch uses three convolution layers, reducing the number of channels to 3. These three channels are used to predict the key position and confidence respectively. The number of channels of the middle convolution layer is 32. The rotation angle prediction is to output a rotation angle through three convolution layers. The coordinates, width and height of the center point output by the network plus the rotation angle constitute the predicted rotation rectangle.

[0030] The step S3 specifically includes:

[0031] S31, each time the courier image is input into the courier detection network based on direction perception, and data enhancement is achieved by means of multi-scale, color space transformation, folding, mosaic, etc., and the loss between the predicted courier position and the real courier position and the key point loss are calculated according to the output of the network. Specific loss functions include ProbIoU loss, classification loss of target detection, DFL (Distribution Focal Loss) loss, regression loss of key points, and confidence loss. Different weights are given to each loss, and then all losses are added together to form the final total loss. The loss function used in the embodiment of the present invention is shown in the following formula:

[0032] L total =L cls +L box +L dfl +L kpts +L conf

[0033] The key point positioning loss uses the OKS loss in YOLOPose. The key point loss is shown in the following formula. In this embodiment, since there is only one key point, the OKS loss function can be simplified to the loss L shown in the following figure kpts , where d n represents the distance between the predicted key point and the true key point, k n represents the weight of the key point, s represents the size of the detected target, δ(v n >0) indicates the visibility of the key point.

[0034]

[0035] S32, using the Adam optimizer, where β1 and β2 are set to 0.9 and 0.9999 respectively, the weight decay is set to 0.0005, the learning initial value is 0.01, and the learning rate is adjusted with a learning rate warm-up and cosine annealing strategy. The number of rounds of learning rate warm-up is set to 3. The learning rate increases linearly in the first three rounds of training, and then the learning rate decays in a cosine manner. At each iteration, the current learning rate and the current loss are calculated, and the loss direction is propagated to update the parameters in the model;

[0036] S33. Repeat the above steps and train for 150 rounds until the loss converges to obtain a trained express delivery order detection model.

[0037] The step S4 specifically includes:

[0038] S41. Scale the image to 640×640 while maintaining the ratio, then normalize the image to between 0 and 1, input the image into the network, obtain the inference result, use the non-maximum suppression of the rotating frame to filter the inferred rotating frame, and obtain the filtered rotating frame and key points. Since the rotating frame is represented by the coordinates of the center point, the width and height of the rotating frame, and the rotation angle, further transformation is required to obtain the final coordinates of the rotating frame. The transformation formula is:

[0039]

[0040] The above formula obtains the coordinates of a vertex of the rotating box, where θ is the rotation angle, w and h are the width and height of the network output respectively, and then the signs in front of the width and height are changed respectively to obtain 4 points. These 4 points are the coordinates of the 4 vertices of the rotating box.

[0041] S42. Arrange the four coordinates clockwise around the center of the rotating frame and connect them in sequence to obtain Figure 5 The rotating box shown in the figure has a detected key point in the upper left corner of the rotating box. By calculating the distance between the key point and the four vertices of the rotating box, the first point in the upper left corner of the express order and the positive direction of the express order are determined, and the coordinates of the other three points are obtained in a clockwise direction. The distance between the coordinates of the first coordinate point and the second coordinate point is used as the width, and the distance between the second coordinate point and the third coordinate point is used as the height. In this way, the correspondence between the four vertices of the original rotating box and the four coordinate points of the positive express order is known. According to the correspondence, the corresponding affine transformation matrix is ​​solved, and the image is cropped from the original image using the affine transformation, and the express order is turned positive, and the positive express order image is directly obtained for subsequent text detection and recognition. Figure 5 The two express delivery orders on the left are the detection results of the model of the present invention. The transformation matrix of the affine transformation is determined by the above algorithm. The rightmost image is the positive express delivery order image obtained after the detection result is affine transformed. Through the above process, it can be seen that the present invention can directly obtain the positive express delivery order image. In the subsequent text recognition process, there is no need to use a direction classifier, which simplifies the recognition steps. The formula of the affine transformation is:

[0042]

[0043] Among them, x and y represent the x and y coordinates of each pixel before the transformation, and the matrix in the middle is the affine transformation matrix. out ,y out Represents the coordinates of the point after the transformation.

[0044] In addition, the model used in this application can achieve an inference speed of 0.016 seconds per test image using RTX 4090. Using YOLOV8s as the backbone network model has a small file size and fast inference speed. The mAP50 of express order detection is 0.99, the mAP50-95 is 0.75, and the mAP50 of key point detection is 0.99. The model size of only 23.9M is also very suitable for deployment on embedded terminal devices.

[0045] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A courier bill visual detection method based on direction perception, characterized in that: The method comprises the following steps: S1, constructing a courier image dataset; S2, building a courier detection model based on direction perception, which is improved based on the existing conventional target detection model, and uses a rotating box detection head and a key point detection head to replace the original horizontal box detection head; S3, training the built courier detection model based on direction perception; S4. Use the trained express delivery order detection model to infer the image containing the express delivery order, obtain the position and key point information of the express delivery order, regard the vertex closest to the key point in the rotation box as the corresponding point of the upper left corner vertex of the upright express delivery order, form 4 groups of corresponding points in clockwise direction, establish an affine transformation matrix, and obtain the forward express delivery order image.

2. The method according to claim 1, characterized in that The step S1 comprises: S11, collecting express delivery order images and integrating them into an initial data set, including express delivery order images taken in different environments and at different resolutions; S12, expanding and annotating the initial data set, using the circumscribed rectangular box of the quadrilateral annotated by the polygon annotation tool as the target box for rotation target detection regression, and using the upper left corner vertex of the express delivery order annotated by the point annotation tool as the key point.

3. The method according to claim 1, characterized in that In the step S2, the constructed courier detection model based on direction perception can be improved based on the existing common horizontal box-based target detection model. Their network structure usually includes three parts: backbone, neck and head. The improved head part, namely the detection head, uses a parallel rotating box detection head and a key point detection head to replace the original horizontal box detection head. In addition to predicting the position of the bounding box and the target category, the rotating box detection head also predicts the rotation angle of the bounding box. The key point detection head is used to locate the upper left corner vertex corresponding to the courier when the courier is upright.

4. The method according to claim 1, characterized in that In the step S3, the training of the constructed courier bill detection model based on direction perception specifically includes: S31, taking a training image from the courier bill image data set each time to the courier bill detection model based on direction perception, inferring the prediction result of the courier bill, calculating the loss between the model prediction result and the sample annotation information, and the loss function of the courier bill visual detection model based on direction perception is composed of the classification loss of target detection, the rotation box regression loss, the key point regression loss and the confidence loss; S32, using the Adam optimizer, the initial learning value is 0.01, and the learning rate is adjusted by using the learning rate preheating and cosine annealing strategy, and the weights of the courier bill detection model based on direction perception are updated by back propagation; S33, repeating the above steps until the model converges to obtain the trained courier bill detection model.

5. The method according to claim 1, characterized in that: The step S4 specifically includes: S41, using the trained express order detection model to infer the image containing the express order, obtain the express order position and key point information, and convert the coordinate representation of the express order position from the coordinates of the center point, width, height and rotation angle to the coordinates of the four vertices of the rotating box; S42, calculating the distances between the four coordinate points and the key points of the rotating rectangular box detected by the express order, and regard the vertex with the smallest distance as the corresponding point of the upper left corner vertex of the upright express order. All vertices form four groups of corresponding points in a clockwise direction, establish an affine transformation matrix, and obtain the forward express order image according to the affine transformation.

Images