A method and device for detecting empty spaces in files based on neural networks

Abstract

This invention relates to a method for detecting text gaps in documents based on a neural network, comprising the following steps: acquiring a text image to be detected, performing a first preprocessing step to obtain a first dataset, and dividing the first dataset into a first training set and a first test set; inputting the first training set into a YOLO V3 neural network for training to generate a trained YOLO V3 neural network detection model; and applying the YOLO V3 neural network detection model. This invention uses acquired text images to form a training set to train the YOLO V3 neural network, and uses the YOLO V3 neural network detection model to detect gaps in text images, achieving high accuracy in text gap recognition.

Description

Technical Field

[0001] This invention belongs to the field of visual recognition technology, and specifically relates to a method and device for detecting empty spaces in files based on neural networks. Background Technology

[0002] As my country places increasing emphasis on industry safety, most manufacturing companies use work permits and other similar documents to record past operations, aiming to improve employee safety awareness and enhance production safety. These documents require the signature of the responsible person for each item during operation. However, these documents often contain numerous clauses and require extensive manual signatures, leading to issues such as missed signatures. Therefore, relevant personnel need to regularly review past documents. However, the sheer volume of these documents and the excessive number of checkpoints required for each document consume significant time and cause inconvenience to relevant staff. Summary of the Invention

[0003] The purpose of this invention is to provide a method and device for detecting empty spaces in files based on neural networks, which can realize the detection of empty spaces in files.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A method for detecting empty spaces in a file based on a neural network includes the following steps:

[0006] The text image to be detected is collected, and after the first preprocessing, the first dataset is obtained. The first dataset is then divided into the first training set and the first test set.

[0007] The first training set is input into the YOLO V3 neural network for training, generating a trained YOLO V3 neural network detection model;

[0008] Application of the YOLO V3 neural network detection model;

[0009] The step of inputting the first training set into the YOLO V3 neural network for training also includes the following steps:

[0010] The first training set is fed into the YOLO V3 neural network in random order, the loss function value of the first training set is calculated, and the weights of the YOLO V3 neural network are updated through an optimization algorithm.

[0011] The first training set is input into the YOLO V3 neural network, and the weights are updated in batches. After each batch, the first test set is fed into the YOLO V3 neural network after the batch update. The loss function value under the current weights is calculated. If the loss function value of the first test set in the current batch is lower than the loss function value of the first training set in the previous batch, the current YOLO V3 neural network weights are saved, and a YOLO V3 neural network detection model is generated.

[0012] After several iterations, the updated YOLO V3 neural network detection model was selected.

[0013] Preferably, the following steps are also included:

[0014] Images of printed and handwritten digits were collected, and after a second preprocessing step, a second dataset was obtained. The second dataset was then divided into a second training set and a second test set.

[0015] The second training set is input into the ResNet18 neural network for training, generating a trained ResNet18 neural network detection model.

[0016] Applications of the ResNet18 neural network detection model;

[0017] The step of inputting the second training set into the ResNet18 neural network for training also includes the following steps:

[0018] The second training set is fed into the ResNet18 neural network in random order, the loss function value of the second training set is calculated, and the weights of the ResNet18 neural network are updated through an optimization algorithm.

[0019] The second training set is fed into the ResNet18 neural network to complete the weight update in batches. After each batch, the second test set is fed into the ResNet18 neural network after the batch update. The loss function value under the current weight is calculated. If the loss function value of the second test set in the current batch is lower than the loss function value of the second training set in the previous batch, the current ResNet18 neural network weights are saved to generate the ResNet18 neural network detection model.

[0020] After several iterations, the updated ResNet18 neural network detection model was selected.

[0021] Preferably, the first preprocessing includes the following steps:

[0022] Filter the captured images and remove blurry or distorted images;

[0023] The remaining images are then subjected to bilinear interpolation to reduce their resolution and unify their resolution.

[0024] Mark the filled and unfilled blank spaces in the remaining images.

[0025] Preferably, the second preprocessing includes the following steps:

[0026] The acquired images are then standardized in resolution using bilinear interpolation.

[0027] After unifying the resolution, the image is processed by grayscale and binarization to remove color factors that have little impact on digit recognition.

[0028] Preferably, the loss function of the YOLO V3 neural network is:

[0029]

[0030] Where Y is the recall rate, the output of the YOLO V3 neural network, α is the balance factor, γ is the adjustment factor, and y represents the category.

[0031] Preferably, the loss function of the ResNet18 neural network is:

[0032]

[0033] Where n represents the number of outputs of the ResNet18 neural network, Y i Identify the i-th output value. The label value that identifies the i-th output value;

[0034] Preferably, the application of the YOLO V3 neural network detection model includes the following steps;

[0035] Acquire the image of the text to be detected and perform the first preprocessing step;

[0036] The input is fed into the YOLO V3 neural network detection model to identify the handwritten numbers and their corresponding printed numbers in the text image to be detected, thus completing the detection of empty spaces in the text image.

[0037] Preferably, the application of the ResNet18 neural network detection model includes the following steps;

[0038] After cutting the handwritten numbers after marking them and the corresponding printed numbers, perform a second preprocessing step.

[0039] The input is fed into the ResNet18 neural network detection model to detect whether the handwritten digits match the corresponding printed digits.

[0040] In another aspect, the present invention also provides a file empty space content detection device based on a neural network, comprising:

[0041] Memory, used to store software applications.

[0042] A processor is used to execute a software application, wherein each program of the software application correspondingly performs the steps in the above-described neural network-based file empty content detection method.

[0043] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described neural network-based file empty space content detection method.

[0044] This invention uses collected images of the text to be detected to form a training set, thereby training a YOLO V3 neural network. It also collects printed and handwritten characters from actual documents to generate a training set, which is used to train a ResNet18 neural network. The text to be detected is then input into the YOLO V3 neural network detection model to detect gaps in the text image. Simultaneously, handwritten digits in the text are identified and cropped to match their corresponding printed digits, and then input into the ResNet18 neural network detection model to check if the handwritten digits match the corresponding printed digits. This invention separates text gap detection from gap digit content recognition, improving the speed and accuracy of text gap detection. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the process structure of a file empty space content detection method based on neural networks according to the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0047] like Figure 1 As shown, a method for detecting empty spaces in a file based on a neural network includes the following steps:

[0048] S10: Collect the text image to be detected, perform the first preprocessing to obtain the first dataset, and divide the first dataset into the first training set and the first test set;

[0049] Specifically, the process involves acquiring images of the text to be detected, including using a camera to capture text images from different angles.

[0050] Specifically, the ratio of the first training set to the first test set is 8:2, in order to train the YOLO V3 neural network;

[0051] In this embodiment, step S10 includes the following steps:

[0052] S11: Filter the captured images and remove blurry or distorted images;

[0053] S12: Perform bilinear interpolation on the remaining images to reduce and unify the image resolution, which facilitates the training of the YOLO V3 neural network.

[0054] S13: Mark the filled and unfilled blank spaces in the remaining images respectively.

[0055] Specifically, in step S13, the Labelme image annotation software is used to annotate the filled and unfilled blank spaces, respectively using solid and dashed boxes or boxes of different colors.

[0056] S20: Input the first training set into the YOLO V3 neural network for training, and generate a trained YOLO V3 neural network detection model;

[0057] Specifically, the YOLO V3 neural network was built using the PyTorch deep learning framework;

[0058] In step S20, the process of inputting the first training set into the YOLO V3 neural network for training also includes the following steps:

[0059] S21: Feed the first training set into the YOLO V3 neural network in random order, calculate the loss function value of the first training set through the loss function of the YOLO V3 neural network, and update the weights of the YOLO V3 neural network through the optimization algorithm;

[0060] S22: Input the first training set into the YOLO V3 neural network, complete the weight update into a batch, and after each batch, send the first test set into the YOLO V3 neural network after the batch update, calculate the loss function value under the current weights, if the loss function value of the first test set in the current batch is lower than the loss function value of the first training set in the previous batch, then save the current YOLO V3 neural network weights and generate the YOLO V3 neural network detection model;

[0061] S23: After several iterations, the updated YOLO V3 neural network detection model is selected.

[0062] Specifically, in step S21, the first training set is fed into the YOLO V3 neural network in random order at a frequency of 2 images at a time, and the loss function value of the current two images is calculated according to the Focal Loss formula. The weights of the YOLO V3 neural network are updated by the Adam optimization algorithm.

[0063] More specifically, in step S21, the loss function of the YOLO V3 neural network is:

[0064]

[0065] Where Y is the recall rate, the output of the YOLO V3 neural network, α is the balance factor, γ is the adjustment factor, and y represents the category.

[0066] Specifically, in step S23, 100 batches of iterations can be selected to choose the latest YOLO V3 neural network detection model. The YOLO V3 neural network detection model can be saved as an ONNX file for easy access in the next step.

[0067] S30: Execution of the YOLO V3 neural network detection model;

[0068] In this embodiment, the application of the YOLO V3 neural network detection model includes the following steps;

[0069] S31: Acquire the image of the text to be detected and perform the first preprocessing;

[0070] S32: Input into the YOLO V3 neural network detection model to identify the handwritten numbers and their corresponding printed numbers in the text image to be detected, thus completing the detection of empty spaces in the text image.

[0071] Specifically, the image is acquired through a camera. After the acquired image undergoes a first preprocessing step, it is sent to the YOLO V3 neural network detection model to detect gaps in the text image. The YOLO V3 neural network then identifies the handwritten digits and their corresponding printed digits, facilitating the next step of detecting the content of the handwritten digits and their corresponding printed digits.

[0072] In general, the text to be checked contains not only text but also some numbers that need to be filled in. These numbers are preceded by corresponding printed numbers. The handwritten numbers must be the same as the corresponding printed numbers. Detecting incorrectly written numbers is one of the problems that needs to be solved.

[0073] This embodiment also includes the following steps:

[0074] S40: Collect images of printed and handwritten digits, perform a second preprocessing step to obtain a second dataset, and divide the second dataset into a second training set and a second test set;

[0075] Specifically, the printed and handwritten numbers collected are derived from numbers in real texts in order to improve the recognition rate in practical applications;

[0076] Specifically, the ratio of the second training set to the second test set is 8:2, which facilitates the training of the ResNet neural network.

[0077] In this embodiment, step S40 includes the following steps:

[0078] S41: Unify the resolution of the acquired images using bilinear interpolation;

[0079] S42: Perform grayscale and binarization processing on the image after unifying the resolution to remove color factors that have little impact on digit recognition.

[0080] Specifically, in step S41, unifying the resolution is to facilitate the training of the ResNet neural network. In step S42, removing color factors that have little impact on digit recognition is to reduce the training difficulty of the ResNet18 neural network and reduce its training time.

[0081] S50: Input the second training set into the ResNet18 neural network for training to generate a trained ResNet18 neural network detection model;

[0082] Specifically, the ResNet18 neural network was built using the PyTorch deep learning framework;

[0083] In step S50, the process of inputting the second training set into the ResNet18 neural network for training also includes the following steps:

[0084] S51: Feed the second training set into the ResNet18 neural network in random order, calculate the loss function value of the current image through the loss function of the ResNet18 neural network, and update the weights of the ResNet18 neural network through the optimization algorithm.

[0085] S52: Feed the second training set into the ResNet18 neural network and complete the weight update in one batch. After each batch, feed the second test set into the ResNet18 neural network updated in the current batch, calculate the loss function value under the current weights, and if the loss function value of the second test set in the current batch is lower than the loss function value of the second test set in the previous batch, save the current ResNet18 neural network weights and generate the ResNet18 neural network detection model.

[0086] S53: After several iterations, the updated ResNet18 neural network detection model is selected.

[0087] Specifically, in step S51, the second training set is fed into the ResNet18 neural network in random order at a frequency of 2 images each time, and the loss function value of the current two images is calculated according to the MSE formula. The weights of the ResNet18 neural network are updated by the Adam optimization algorithm.

[0088] More specifically, in step S51, the ResNet18 neural network selects the Mean Square Error (MSE) formula as its loss function. The loss function of the ResNet18 neural network is:

[0089]

[0090] Where n represents the number of outputs of the ResNet18 neural network, Y i Identify the i-th output value. The label value that identifies the i-th output value;

[0091] Specifically, in step S53, 100 batches of iterations can be selected to choose the latest ResNet18 neural network detection model. The ResNet18 neural network detection model can be saved as an ONNX file for easy access in the next step.

[0092] S60: Application of ResNet18 neural network detection model;

[0093] In this embodiment, the application of the ResNet18 neural network detection model includes the following steps;

[0094] S61: Cut the handwritten digits after marking them and the corresponding printed digits, and then perform a second preprocessing on them;

[0095] S62: Input into the ResNet18 neural network detection model to detect whether the handwritten digits match the corresponding printed digits.

[0096] Specifically, the handwritten digits identified in step S32 are cut off from their corresponding printed digits. After the second preprocessing, the cut-off images and detection results are output to the computer screen.

[0097] In the above scheme, image acquisition is performed using a camera. After the acquired image undergoes a first preprocessing step, it is fed into the YOLO V3 neural network detection model to detect gaps in the text image. Then, the handwritten digits are marked with their corresponding printed digits. The marked handwritten digits and their corresponding printed digits are cut off, and after a second preprocessing step, they are input into the ResNet18 neural network detection model to detect whether the handwritten digits and their corresponding printed digits are consistent. Finally, the text image and the gap detection results are output to the computer screen. At the same time, the image of the cut handwritten digits and their corresponding printed digits, along with the detection results, are also output to the computer screen to assist the staff in completing the inspection.

[0098] In another embodiment, if only handwritten numbers and their corresponding printed numbers are on the same horizontal line in the text to be detected, then the YOLO V3 neural network is used to mark all empty spaces and printed numbers in the text, mark the coordinates of the characters, and then judge the coordinates. If two coordinates are on the same horizontal line, then the two characters are cut off, and after the second preprocessing, they are input into the called ResNet18 neural network detection model to determine whether the two characters are consistent.

[0099] According to another aspect of the present invention, a file empty content detection device based on a neural network is also provided, comprising: a memory for storing a software application, and a processor for executing the software application, wherein each program of the software application correspondingly performs the steps in the method of the present invention.

[0100] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the method of the present invention.

Claims

1. A method for detecting empty spaces in a file based on a neural network, characterized in that, Includes the following steps: The text image to be detected is collected, and after the first preprocessing, the first dataset is obtained. The first dataset is then divided into the first training set and the first test set. The first training set is input into the YOLO V3 neural network for training, generating a trained YOLO V3 neural network detection model; Application of the YOLO V3 neural network detection model; The step of inputting the first training set into the YOLO V3 neural network for training also includes the following steps: The first training set is fed into the YOLO V3 neural network in random order, the loss function value of the first training set is calculated, and the weights of the YOLO V3 neural network are updated through an optimization algorithm. The first training set is input into the YOLO V3 neural network, and the weights are updated in batches. After each batch, the first test set is fed into the YOLO V3 neural network after the batch update. The loss function value under the current weights is calculated. If the loss function value of the first test set in the current batch is lower than the loss function value of the first training set in the previous batch, the current YOLO V3 neural network weights are saved, and a YOLO V3 neural network detection model is generated. After several iterations, the updated YOLO V3 neural network detection model was selected. It also includes the following steps: Images of printed and handwritten digits were collected, and after a second preprocessing step, a second dataset was obtained. The second dataset was then divided into a second training set and a second test set. The second training set is input into the ResNet18 neural network for training, generating a trained ResNet18 neural network detection model. Applications of the ResNet18 neural network detection model; The step of inputting the second training set into the ResNet18 neural network for training also includes the following steps: The second training set is fed into the ResNet18 neural network in random order, the loss function value of the second training set is calculated, and the weights of the ResNet18 neural network are updated through an optimization algorithm. The second training set is fed into the ResNet18 neural network, and the weights are updated in batches. After each batch, the second test set is fed into the ResNet18 neural network after the batch update. The loss function value under the current weights is calculated. If the loss function value of the second test set in the current batch is lower than the loss function value of the second training set in the previous batch, the current ResNet18 neural network weights are saved, and a ResNet18 neural network detection model is generated. After several iterations, the updated ResNet18 neural network detection model was selected. The process involves first preprocessing the acquired images, then feeding them into the YOLO V3 neural network detection model to detect gaps in the text image. Next, the handwritten digits are matched with their corresponding printed digits. The matched handwritten digits and their corresponding printed digits are then cut and processed a second time before being fed into the ResNet18 neural network detection model to check for consistency between the handwritten digits and their corresponding printed digits. Finally, the text image and gap detection results are output to the computer screen, along with the cut image of the handwritten digits and their corresponding printed digits, and the detection results, to assist staff in the inspection process.

2. The method for detecting empty spaces in a file based on a neural network according to claim 1, characterized in that, The first preprocessing includes the following steps: Filter the captured images and remove blurry or distorted images; The remaining images are then subjected to bilinear interpolation to reduce their resolution and unify their resolution. Mark the filled and unfilled blank spaces in the remaining images.

3. The method for detecting empty spaces in a file based on a neural network according to claim 1, characterized in that, The second preprocessing includes the following steps: The acquired images are then standardized in resolution using bilinear interpolation. After unifying the resolution, the image is processed by grayscale and binarization to remove color factors that have little impact on digit recognition.

4. The method for detecting empty spaces in a file based on a neural network according to claim 1, characterized in that, The loss function of the YOLO V3 neural network is: ， Where Y is the recall rate, the output of the YOLO V3 neural network, α is the balance factor, γ is the adjustment factor, and y represents the category.

5. The method for detecting empty spaces in a file based on a neural network according to claim 1, characterized in that, The loss function of the ResNet18 neural network is: ， Where n represents the number of outputs of the ResNet18 neural network. Identify the i-th output value. The label value that identifies the i-th output value.

6. A file space content detection device based on neural networks, characterized in that, include: Memory, used to store software applications. A processor for executing a software application, wherein each program of the software application correspondingly performs the steps of the neural network-based file space content detection method as described in claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it performs the steps of the neural network-based file empty content detection method as described in any one of claims 1 to 5.

Images