CRNN-based ship character recognition method in natural scene
By combining Cyclegan and Pix2pix models for image restoration, the problem of low accuracy in ship text recognition in natural scenes is solved, efficient recognition under fuzzy and occlusion is achieved, and model training needs with small data sample size is met.
Patent Information
- Application Number
- CN202411880259.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-04-15
AI Technical Summary
The prior art is difficult to efficiently identify ship text in natural scenarios, especially in the case of blur, occlusion and small data sample size, the recognition accuracy is low.
The Cyclegan model is combined with the Pix2pix model to restore images and improve text recognition accuracy. The specific steps include collecting ship scene pictures, text detection, image convolution, blur and occlusion detection and restoration, building and training the CRNN model, and finally performing ship text recognition.
Through image restoration technology, the training sample size of CRNN models has been reduced from tens of thousands to less than 4000, which has improved the recognition accuracy, especially in the case of blur and occlusion, and the recognition success rate has been significantly improved.
Smart Images
Figure CN119832534A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of ship natural scene text recognition, and in particular to a ship text recognition method in a natural scene based on CRNN. Background Art
[0002] So far, my country mainly relies on the AIS system to identify ships in waterways. However, for some illegal cargo ships or when the AIS system fails, we can only rely on manual observation. However, this method is prone to missed detection and misjudgment in river basins with large flow rates, and is inefficient. Therefore, we use deep learning and image processing technology to correct and identify the text on the ship's side.
[0003] Recognizing text information in natural scene images has received increasing attention in computer vision. Text recognition is a part of OCR (Optical Character Recognition). Optical character recognition mainly consists of two parts: text detection and text recognition. The main difficulties in text recognition in natural scene images come from the following factors.
[0004] 1. The scene text has great changes in character fonts, colors and languages.
[0005] 2. Most scene images experience uneven lighting, blur, perspective distortion, low contrast, low resolution, and occlusion.
[0006] 3. Text lines in the wild may have irregular shapes, such as curved shapes.
[0007] In recent years, thanks to the development of deep learning, conventional text recognition has made significant progress. Existing automatic text recognition methods have a high recognition accuracy for neat text, but the recognition accuracy is low for text in natural scenes such as text on the side of a ship in a river, which is prone to deformation, wear and tear, and occlusion.
[0008] In addition, improvements based on text recognition models often require a large data set of more than 10,000 to retrain the recognition model, which is difficult for relatively small data such as the text on the side of river ships, which has a sample size of only a few hundred or so.
[0009] In summary, the existing methods cannot complete the task of ship side text recognition well and have great limitations. Summary of the invention
[0010] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a method for ship text recognition in natural scenes based on CRNN. The method for ship text recognition in natural scenes based on CRNN combines the Cyclegan model with the Pix2pix model, which can restore the image and improve the accuracy of text recognition when the image is blurred or occluded.
[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0012] A method for ship text recognition in natural scenes based on CRNN includes the following steps.
[0013] Step 1, collecting ship scene pictures: collect N pictures of ship scenes with known identities in the waterway; where 100≤N≤1000; the N pictures of ship scenes include text blurred pictures and text blocked pictures.
[0014] Step 2: Text detection: Detect target text in each ship scene image collected in step 1 and crop the target text image.
[0015] Step 3: Image convolution: Perform convolution on each target text image obtained in step 2.
[0016] Step 4: Blur detection: Perform blur detection on each target text image after convolution processing to obtain M blurred images and NM non-blurred images.
[0017] Step 5: Restoration of blurred images: Each blurred image is restored using the Cyclegan model to obtain M blurred restored images.
[0018] Step 6, occlusion detection: Perform occlusion detection on the NM non-blurred images obtained in step 4 and the M blurred restored images obtained in step 5 to obtain A occluded images and NA non-occluded images.
[0019] Step 7: Occlusion restoration: Each occluded image is restored using the Pix2pix model to obtain A occluded restored images.
[0020] Step 8. Build and train the CRNN model: Use the NA non-occluded images obtained in step 6 and the A occluded and restored images obtained in step 7 as the input of the CRNN model, and use the ship identities corresponding to the N ship scene images in step 1 as the output, and then train the CRNN model.
[0021] Step 9, ship text recognition: collect scene pictures of unknown ships to be detected in the waterway, repeat steps 2 to 7 to obtain fuzzy occlusion restoration pictures, and then input the fuzzy occlusion restoration pictures into the CRNN model trained in step 8 to obtain the identity information of the ship to be detected.
[0022] In step 1, the text-blocked image is caused by the text on the ship's side being blocked by the cables of the ship's hull, and the text-blocked image includes an image in which the cables obliquely block the text on the ship's side.
[0023] In step 1, 100≤N≤400.
[0024] In step 2, the YOLOv5 detection model is used to detect target text in each ship scene image.
[0025] In step 4, each target text image after convolution processing is blurred by using the Laplacian operator.
[0026] In step 4, the Laplacian operator uses a 3*3 discrete convolution kernel for blur detection, specifically:
[0027]
[0028] In step 6, the GAN discriminator is used to perform occlusion detection on the non-blurred image and the blurred restored image.
[0029] The GAN discriminator uses the mean square loss function MSELoss to perform convergence judgment of occlusion; the expression of the mean square loss function MSELoss is:
[0030]
[0031] Where W is the number of regions to be identified; x i is the gray value of the i-th discrimination area in the input image to be discriminated; y i is the grayscale value of the i-th discrimination area in the benchmark occluded image; the occluded area in the benchmark occluded image is not less than 20%, and the initial recognition confidence of the occluded text should be no higher than 70%. In step 7, the loss function of the Pix2pix model has a hyperparameter λ, which restores the same test image by setting different hyperparameters and selects the hyperparameter value when the recognition confidence is maximum.
[0032] In step 7, different hyperparameters are set to 5, 10, 20, 25 and 30 respectively, and the set hyperparameter 20 corresponding to the maximum recognition confidence is selected as the hyperparameter λ of the loss function of the Pix2pix model.
[0033] The present invention has the following beneficial effects:
[0034] 1. The present invention combines the Cyclegan model with the Pix2pix model to restore blurred and occluded images, thereby reducing the training sample size of the CRNN model from tens of thousands to less than 400 without affecting the training effect, thereby meeting the model training and efficient text recognition requirements of river ship side text with a relatively small sample size.
[0035] 2. The present invention can restore fuzzy, blocked and missing characters with a low recognition success rate and obtain correct recognition results. Compared with traditional character recognition methods, the present invention has higher recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 The flowchart of the ship text recognition method in natural scene based on CRNN of the present invention is shown.
[0037] Figure 2 A flow chart of the YOLOv5 detection model used in the present invention is shown.
[0038] Figure 3 The flowchart of the CRNN model used in the present invention is shown.
[0039] Figure 4 The figure shows the results of restoring the image using the Cyclegan model, where the left side (a) shows blurred text and the right side (b) shows clear text restored from blurred text.
[0040] Figure 5 The figure shows the results of text restoration using the Pix2pix model, where the left side (a) shows the occluded text and the right side (b) shows the occluded and restored text. DETAILED DESCRIPTION
[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific preferred embodiments.
[0042] In the description of the present invention, it should be understood that the terms "left side", "right side", "upper part", "lower part" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. "First", "second" and the like do not indicate the importance of the components, and therefore cannot be understood as limiting the present invention. The specific dimensions used in this embodiment are only for illustrating the technical solution, and do not limit the scope of protection of the present invention.
[0043] like Figure 1 As shown, a ship text recognition method in natural scenes based on CRNN includes the following steps.
[0044] Step 1, collecting ship scene pictures: collect N pictures of ship scenes with known identities in the waterway; wherein, 100≤N≤1000, preferably 100≤N≤400, and preferably N=300-400 in this embodiment; the N pictures of the ship scene include text-blurred pictures and text-occluded pictures.
[0045] By adopting steps 2 to 7, the present invention can restore blurred and occluded images, thereby reducing the training sample size of the CRNN model from tens of thousands to less than 400 without affecting the training effect, thereby meeting the model training and efficient text recognition requirements of river ship side text with a relatively small sample size.
[0046] The applicant found in actual investigation that the text on the side of river vessels is mainly blocked by the cables on the hull. The thickness and blocking method of the cables are random. According to experiments and observations, the accuracy of text recognition is most affected when the cables block the text obliquely. Therefore, when selecting training images, it is preferred to select text images with oblique blocking, or artificial processing is performed to imitate real blocking. Therefore, the above text blocking images include images of cables blocking the text on the side of the ship obliquely.
[0047] Step 2: Text detection: Figure 2 As shown, it is preferred to use the YOLOv5 detection model to detect the target text on each ship scene picture collected in step 1, and crop the target text picture.
[0048] Step 3: Image convolution: Perform convolution on each target text image obtained in step 2.
[0049] Step 4, blur detection: For each target text image after convolution processing, the Laplace operator is preferably used to perform blur detection to obtain M blurred images and NM non-blurred images.
[0050] The expression of image clarity D(f) based on the Laplace operator is:
[0051] D(f)=∑y∑x|G(x,y)| G(x,y)>T
[0052] Where G(x, y) is the convolution of the Laplace algorithm at the pixel point (x, y), and T is the set edge detection threshold.
[0053] When D(f)≥D0, it is judged that the ship scene image to be identified does not have blurring caused by deformation or wear, and directly proceeds to step 6; otherwise, it is judged that the current ship scene image is a blurred image, and it is necessary to proceed to step 5.
[0054] Furthermore, the above Laplacian operator uses a 3*3 discrete convolution kernel during blur detection, specifically:
[0055]
[0056] Step 5: Restoration of blurred images: Each blurred image is restored using the Cyclegan model to obtain M blurred restored images.
[0057] For the Cyclegan model:
[0058] First, a generator model based on the GAN network from X→Y is established. The loss function of the model is:
[0059]
[0060] where D X is the discriminator for X, which needs to be trained to maximize it. G represents the image generator from X→Y, and G(x) is the Y generated from X.
[0061] At the same time, there must be a generator model based on the GAN network from Y→X, and the loss function of the model is:
[0062]
[0063] where D Y is the discriminator for Y, which needs to be trained to maximize it. F represents the image generator from Y→X, and F(y) is X generated from Y. When starting from sample X of X, the reconstructed result F(G(x)) should be as close to the original image X as possible. The closeness between X and F(G(x)) is usually measured by L1 or L2 norm. At the same time, another mapping from Y to G(F(y)) should also maintain the same criteria. The cycle consistency of Cyclegan can be expressed as:
[0064]
[0065] The loss function of the entire Cyclegan is:
[0066]
[0067] Among them, λ′ is a hyperparameter used to balance cycle consistency and edge matching.
[0068] The present invention uses the Cyclegan model to restore the pictures before and after, such as Figure 4 shown.
[0069] Step 6, occlusion detection: The NM non-blurred images obtained in step 4 and the M blurred restored images obtained in step 5 are preferably subjected to occlusion detection using a GAN discriminator to obtain A occluded images and NA non-occluded images.
[0070] The above-mentioned GAN discriminator preferably uses the mean square loss function MSELoss to perform convergence judgment of occlusion; wherein the expression of the mean square loss function MSELoss is:
[0071]
[0072] Where W is the number of regions to be identified; x i is the grayscale value of the i-th discrimination area in the input image to be judged; yi is the grayscale value of the i-th discrimination area in the reference occluded image; the occluded area in the reference occluded image is not less than 20%, and the initial recognition confidence of the occluded text should not be higher than 70%.
[0073] Step 7: Occlusion restoration: Each occluded image is restored using the Pix2pix model to obtain A occluded restored images.
[0074] The Pix2pix model is a cGAN-based image restoration network. First, a cGAN is trained to map a contour image to a photo. The discriminator D learns to classify fake images (images synthesized by the generator) and real images. At the same time, the generator G must learn to deceive the discriminator. The discriminator D in this model is a discriminator called Patchgan. In terms of loss function, the loss function of the pix2pix model also borrows from the loss function of cGAN:
[0075]
[0076] The whole formula consists of two terms, x represents the source domain image, y represents the real image, z represents the noise input to the G network, and G(x, z) represents the target domain image generated by the G network based on the source domain image and random noise.
[0077] D(x, y) represents the probability that the D network judges whether the real picture is real, while D(x, G(x, z)) is the probability that the D network judges whether the picture generated by G is real.
[0078] Therefore, the final loss function value G* of the Pix2pix model is expressed as:
[0079]
[0080] In the formula, λ is a hyperparameter. In the prior art, a fixed value of 10 is usually selected. In the usage scenario data in this application, it is found that λ=10 cannot achieve a good restoration effect. Therefore, by setting different hyperparameters λ, such as 5, 10, 20, 25 and 30, the same test image is restored, and the hyperparameter value at the maximum recognition confidence is selected. In this embodiment, it is found through repeated training that when λ=20, the Pix2pix model has the best restoration effect and has the maximum recognition confidence.
[0081] The present invention uses the Pix2pix model to restore the photos before and after occlusion. Figure 5 shown.
[0082] Step 8. Build and train the CRNN model: Use the NA non-occluded images obtained in step 6 and the A occluded and restored images obtained in step 7 as the input of the CRNN model, and use the ship identities corresponding to the N ship scene images in step 1 as the output, and then train the CRNN model.
[0083] The network structure of the CRNN model is as follows: Figure 3 As shown, the prior art includes a convolutional layer, a recurrent layer, and a transcription layer.
[0084] Step 9, ship text recognition: collect scene pictures of unknown ships to be detected in the waterway, repeat steps 2 to 7 to obtain fuzzy occlusion restoration pictures, and then input the fuzzy occlusion restoration pictures into the CRNN model trained in step 8 to obtain the identity information of the ship to be detected.
[0085] The present invention can restore fuzzy, blocked and missing characters with a low recognition success rate and obtain correct recognition results. Compared with traditional character recognition methods, the present invention has higher recognition accuracy.
[0086] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.
Claims
1. A ship text recognition method in natural scenes based on CRNN, characterized by: The steps include: Step 1, collecting ship scene pictures: collecting N pictures of ship scenes with known identities in the waterway; wherein, 100≤N≤1000; the N pictures of ship scenes include text-blurred pictures and text-occluded pictures; Step 2, text detection: Detect the target text on each ship scene image collected in step 1, and crop the target text image; Step 3, image convolution: perform convolution processing on each target text image obtained in step 2; Step 4, blur detection: perform blur detection on each target text image after convolution processing to obtain M blur images and NM non-blur images; Step 5, blur image restoration: Each blur image is restored using the Cyclegan model to obtain M blur restored images; Step 6, occlusion detection: perform occlusion detection on the NM non-blurred images obtained in step 4 and the M blurred restored images obtained in step 5 to obtain A occluded images and NA non-occluded images; Step 7, occlusion restoration: each occluded image is restored using the Pix2pix model to obtain A occluded restored images; Step 8, construct and train the CRNN model: use the NA non-occluded images obtained in step 6 and the A occluded restored images obtained in step 7 as inputs to the CRNN model, use the ship identities corresponding to the N ship scene images in step 1 as outputs, and then train the CRNN model; Step 9, ship text recognition: collect scene pictures of unknown ships to be detected in the waterway, repeat steps 2 to 7 to obtain fuzzy occlusion restoration pictures, and then input the fuzzy occlusion restoration pictures into the CRNN model trained in step 8 to obtain the identity information of the ship to be detected.
2. According to the CRNN-based ship text recognition method in natural scenes according to claim 1, it is characterized by: In step 1, the text-blocked image is caused by the text on the ship's side being blocked by the cables of the ship's hull, and the text-blocked image includes an image in which the cables obliquely block the text on the ship's side.
3. The method for ship text recognition in natural scenes based on CRNN according to claim 1 is characterized by: In step 1, 100≤N≤400.
4. The method for ship text recognition in natural scenes based on CRNN according to claim 1 is characterized in that: In step 2, the YOLOv5 detection model is used to detect target text in each ship scene image.
5. The method for ship text recognition in natural scenes based on CRNN according to claim 1 is characterized in that: In step 4, each target text image after convolution processing is blurred by using the Laplacian operator.
6. The method for ship text recognition in natural scenes based on CRNN according to claim 5 is characterized by: In step 4, the Laplacian operator uses a 3*3 discrete convolution kernel for blur detection, specifically:
7. The method for ship text recognition in natural scenes based on CRNN according to claim 1 is characterized by: In step 6, the GAN discriminator is used to perform occlusion detection on the non-blurred image and the blurred restored image.
8. The method for ship text recognition in natural scenes based on CRNN according to claim 7 is characterized by: The GAN discriminator uses the mean square loss function MSELoss to perform convergence judgment of occlusion; the expression of the mean square loss function MSELoss is: Where W is the number of regions to be identified; x i is the gray value of the i-th discrimination area in the input image to be discriminated; y i is the grayscale value of the ith discrimination area in the reference occluded image; the occluded area in the reference occluded image is not less than 20%, and the initial recognition confidence of the occluded text should be no higher than 70%.
9. The method for ship text recognition in natural scenes based on CRNN according to claim 1 is characterized by: In step 7, the loss function of the Pix2pix model has a hyperparameter λ, which restores the same test image by setting different hyperparameters and selects the hyperparameter value when the recognition confidence is maximum.
10. The method for ship text recognition in natural scenes based on CRNN according to claim 9, characterized in that: In step 7, different hyperparameters are set to 5, 10, 20, 25 and 30 respectively, and the set hyperparameter 20 corresponding to the maximum recognition confidence is selected as the hyperparameter λ of the loss function of the Pix2pix model.