Image main body automatic identification system and segmentation method

Abstract

The invention relates to the technical field of image data processing, in particular to an image main body automatic identification system and segmentation method, and the system comprises terminal equipment, an identification segmentation platform and a database. The terminal equipment is handheld electronic equipment, is physical equipment used for connecting a user and the identification segmentation platform in a communication network, and comprises an image acquisition unit, a display touch control unit and a communication unit; the segmentation method of the image main body automatic identification system comprises the following steps: S1, shooting a test question image through a terminal device, and transmitting the test question image to an identification segmentation platform; s2, preprocessing the image; s3, area identification; S4, marking the divided areas, returning the marked areas to the terminal equipment, and receiving a mark number selected by a user; and S6, text information of the segmented areas is extracted through an OCR function, similarity matching is carried out on the text information and the database, and a matching result is returned. The method has the advantages of high-precision region division, algorithm robustness and quick retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to an image subject automatic recognition system and a segmentation method. Background Art

[0002] With the advancement of network technology, online question searching has become popular. In general, the technology is to collect test information through terminal devices (such as mobile phones, tablets, etc.) and upload it to the cloud server. The server analyzes and processes the input information, and after comparing it with the massive exercise materials in the database, the highly matched results are returned to the terminal to complete the question search. The collection of test information generally requires shooting, so an automatic image subject recognition system is needed to identify the test information in the picture and segment the test information.

[0003] Existing automatic image subject recognition systems usually recognize text and automatically segment the image area through a raster method to separate the test question information. However, since a large amount of test question text is generally densely arranged together, this recognition and segmentation method is difficult to accurately identify which part of the question the student is searching for in the image, and thus cannot specifically segment the required search part. Therefore, based on the above problems, an automatic image subject recognition system and segmentation method are proposed. Summary of the invention

[0004] The purpose of the present invention is to provide an image subject automatic recognition system and segmentation method to solve the problem that the existing image subject automatic recognition system is difficult to accurately identify which part of the title the user wants to search for from a large amount of text in the image, and is unable to specifically segment the required search part.

[0005] To achieve the above object, the present invention provides the following technical solutions: In a first aspect, the present invention provides an automatic image subject recognition system, including a terminal device, a recognition and segmentation platform, and a database; The terminal device is a handheld electronic device, comprising: An image acquisition unit, used to capture and convert images of the real world into digital signals; A display touch unit, used to display images and receive touch selection instructions from users; A communication unit, used for transmitting data with the identification and segmentation platform; The identification and segmentation platform includes: A transmission unit, used for communicating with terminal equipment and database; A pre-processing unit, used to perform white balance, brightness compensation, grayscale and high-resolution sharpening processing on the image; The region recognition module includes a text recognition unit, a gap recognition unit and a division unit, wherein the text recognition unit is used to recognize the title number at the beginning of the text paragraph, the gap recognition unit is used to analyze the paragraph spacing difference, and the division unit divides the title region based on the title number or the spacing feature; The marking unit is used to mark the divided areas and transmit them back to the terminal device; A yes or no selection unit, used to receive the label selected by the user and trigger the segmentation instruction; A segmentation unit, used to perform mask segmentation on non-target areas according to user selection; The information extraction unit uses OCR technology to extract text information from the segmented area; A retrieval unit, used to match the extracted topic information with the database and return the result; The database stores exercise data and is connected to the retrieval unit through the transmission unit.

[0006] In a second aspect, the present invention further provides a segmentation method based on an automatic image subject recognition system, comprising the following steps: S1. Take a picture of the test question through the terminal device and transmit it to the recognition and segmentation platform; S2. performing preprocessing on the image, including brightness compensation based on histogram equalization, white balance adjustment based on grayscale world assumption, grayscale, and high-resolution sharpening using Laplacian operator; S3. Region Identification: If a title number is detected at the beginning of a text paragraph, the title number is located through a template matching algorithm, and the region is divided using the title number as the boundary; If there is no title number, the text line boundary is extracted through Canny edge detection, and the paragraph spacing is calculated in combination with the dynamic threshold algorithm of the gap recognition unit. If the spacing is greater than the preset threshold, the area is divided using the gap as the dividing line; Gaussian filter standard deviation of the Canny edge detector σ is 1.0, and the ratio of the high threshold to the low threshold is 3:1; S4 will be divided into regional labels and sent back to the terminal device, receiving the user selected label; S5. Segmentation processing: Generates a binary mask based on user selection and applies morphological closing operations to fill holes; The watershed algorithm is used to accurately segment the mask area and retain the target area; S6. Extract the text information of the segmented area through the OCR function, perform similarity matching with the database, and return the matching result.

[0007] Furthermore, the brightness compensation based on histogram equalization in step S2 has an equalization function as follows: ; Among them, v is the current pixel gray value, ni Grayscale i The number of pixels, N is the total number of pixels, L max is the maximum grayscale level, default L max=255.

[0008] Furthermore, the dynamic threshold algorithm formula in the region identification in step S3 is: Define spacing thresholds T = μ + kσ , in μ is the average spacing, σ is the standard deviation, k is the empirical coefficient, k The value range is 1.2≤ k ≤2.0, default k =1.5, used to distinguish the gaps between questions.

[0009] Furthermore, the watershed algorithm in the segmentation process of step S5 generates a marked area based on the gradient image and separates the adhesion area through the flooding model. The function expression is: Watershed I grad,Markers) in I Grad is the preprocessed gradient image, and Markers is the initial marker of the user-selected area.

[0010] Furthermore, the morphological closing operation in step S5 uses a 3×3 rectangular kernel, whose structural element is defined as: ; And the number of closing operation iterations is 1.

[0011] Furthermore, the gradient image of the watershed algorithm in step S5 is calculated by the Sobel operator, and its horizontal and vertical gradient weights are respectively: ; And the gradient amplitude is calculated as ;in Gx is the horizontal gradient, Gy is the vertical gradient, G is the gradient amplitude.

[0012] Furthermore, the OCR function in step S6 uses a convolutional neural network (CNN) for text detection and combines it with a bidirectional LSTM for sequence recognition to output the text sequence with the highest probability.

[0013] Furthermore, in step S6, the convolutional neural network CNN performs text detection, using a ResNet-50 backbone network: the input image is normalized to 32×256×332×256×3, and multi-scale features are extracted through the convolution layer of ResNet-50; The output feature map generates candidate boxes through RPN. The anchor points of RPN are set to different aspect ratios (0.5, 1, 2) and the scale is [8, 16, 32][8, 16, 32] pixels. Non-maximum suppression (NMS): Filter overlapping candidate boxes, suppress redundant detection, set the threshold to 0.7, and ensure that only one optimal candidate box is retained for each text line.

[0014] Furthermore, the sequence recognition in step S6 is a bidirectional LSTM and attention mechanism: A bidirectional LSTM network is used, which includes 2 hidden layers, and the number of hidden units in each layer is 256. The LSTM input is the text line image features output by the detection module. The feature extraction is realized through the RoI Align layer, and the output feature map is a fixed size of 8×328×32. Combined with the attention mechanism, the context vector of each time step is calculated using the formula: ; in, is the context vector for each time step, ht is the LSTM hidden state, is the context vector of the previous time step, Wa is the trainable weight matrix.

[0015] The beneficial effects of the present invention are as follows: the automatic image subject recognition system and segmentation method of the present invention have the following advantages: High-precision area segmentation: Combine title number detection with dynamic gap threshold to solve the problem of dense text segmentation.

[0016] User interaction optimization: label frame selection and touch selection mechanism to improve the efficiency of target area positioning.

[0017] Algorithm robustness: The watershed algorithm is combined with morphological closing operation to avoid over-segmentation or under-segmentation.

[0018] Fast retrieval: Efficient matching of OCR models and databases to achieve search response within seconds. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the system structure of the present invention; Figure 2This is a schematic diagram of the core code for image preprocessing of the present invention; Figure 3 This is a schematic diagram of the core code for regional division and segmentation of the present invention; Figure 4 This is a schematic diagram of the core code for OCR text extraction of the present invention; Figure 5 This is a schematic diagram of traditional grid segmentation; Figure 6 Schematic diagram for comparison of grid segmentation of the present invention. DETAILED DESCRIPTION

[0020] See also Figure 1 , the present invention provides a technical solution: Embodiment 1, as Figure 1 As shown, an automatic image subject recognition system includes a terminal device, a recognition and segmentation platform and a database; The terminal device is a handheld electronic device, comprising: An image acquisition unit, used to capture and convert images of the real world into digital signals; A display touch unit, used to display images and receive touch selection instructions from users; A communication unit, used for transmitting data with the identification and segmentation platform; The identification and segmentation platform includes: A transmission unit, used for communicating with terminal equipment and database; A pre-processing unit, used to perform white balance, brightness compensation, grayscale and high-resolution sharpening processing on the image; The region recognition module includes a text recognition unit, a gap recognition unit and a division unit, wherein the text recognition unit is used to recognize the title number at the beginning of the text paragraph, the gap recognition unit is used to analyze the paragraph spacing difference, and the division unit divides the title region based on the title number or the spacing feature; The marking unit is used to mark the divided areas and transmit them back to the terminal device; A yes or no selection unit, used to receive the label selected by the user and trigger the segmentation instruction; A segmentation unit, used to perform mask segmentation on non-target areas according to user selection; The information extraction unit uses OCR technology to extract text information from the segmented area; A retrieval unit, used to match the extracted topic information with the database and return the result; The database stores exercise data and is connected to the retrieval unit through the transmission unit.

[0021] Embodiment 2, as Figure 2-Figure 6 As shown, a segmentation method based on an automatic image subject recognition system comprises the following steps: S1. Take a picture of the test question through the terminal device and transmit it to the recognition and segmentation platform; S2. performing preprocessing on the image, including brightness compensation based on histogram equalization, white balance adjustment based on grayscale world assumption, grayscale, and high-resolution sharpening using Laplacian operator; The user takes a picture of the test paper, and the system performs brightness compensation and edge enhancement. The brightness compensation based on histogram equalization has an equalization function as follows: ; Among them, v is the current pixel gray value, ni Grayscale i The number of pixels, N is the total number of pixels, L max is the maximum grayscale level, default L max=255.

[0022] The core code can be found in Figure 2 As shown: S3. Region Identification: If a title number is detected at the beginning of a text paragraph, the title number is located through a template matching algorithm, and the region is divided using the title number as the boundary; If there is no title number, the text line boundary is extracted through Canny edge detection, and the paragraph spacing is calculated in combination with the dynamic threshold algorithm of the gap recognition unit. If the spacing is greater than the preset threshold, the area is divided using the gap as the dividing line; Gaussian filter standard deviation of the Canny edge detector σ is 1.0, and the ratio of the high threshold to the low threshold is 3:1.

[0023] The dynamic threshold algorithm formula in the region identification in step S3 is: Define spacing thresholds T = μ + kσ , in μ is the average spacing, σ is the standard deviation, k is the empirical coefficient, k The value range is 1.2≤ k ≤2.0, default k =1.5, used to distinguish the gaps between questions.

[0024] like Figure 5 As shown, the traditional method combines 3 questions into 1 area; Figure 6 As shown: the present invention is correctly divided into three areas.

[0025] Its core code is as follows Figure 3 As shown: S4 will be divided into regional labels and sent back to the terminal device, receiving the user selected label; S5. Segmentation processing: Generates a binary mask based on user selection and applies morphological closing operations to fill holes; The watershed algorithm is used to accurately segment the mask area and retain the target area; The watershed algorithm in the segmentation process of step S5 generates a marked area based on the gradient image and separates the adhesion area through the flooding model. The function expression is: Watershed I grad,Markers) in I Grad is the preprocessed gradient image, and Markers is the initial marker of the user-selected area.

[0026] The morphological closing operation uses a 3×3 rectangular kernel, whose structural element is defined as: ; And the number of closing operation iterations is 1.

[0027] The gradient image of the watershed algorithm is calculated by the Sobel operator, and its horizontal and vertical gradient weights are: ; And the gradient amplitude is calculated as ;in Gx is the horizontal gradient, Gy is the vertical gradient, G is the gradient amplitude.

[0028] S6. Extract the text information of the segmented area through the OCR function, perform similarity matching with the database, and return the matching result. The core code is as follows: Figure 4 shown.

[0029] In step S6, the convolutional neural network CNN is used for text detection, and the ResNet-50 backbone network is used: the input image is normalized to 32×256×332×256×3, and multi-scale features are extracted through the convolution layer of ResNet-50; The output feature map generates candidate boxes through RPN. The anchor points of RPN are set to different aspect ratios (0.5, 1, 2) and the scale is [8, 16, 32][8, 16, 32] pixels. Non-maximum suppression (NMS): Filter overlapping candidate boxes, suppress redundant detection, set the threshold to 0.7, and ensure that only one optimal candidate box is retained for each text line.

[0030] The sequence recognition in step S6 is a bidirectional LSTM and attention mechanism. The core code is as follows Figure 4 As shown: A bidirectional LSTM network is used, which includes 2 hidden layers, and the number of hidden units in each layer is 256. The LSTM input is the text line image features output by the detection module. The feature extraction is realized through the RoI Align layer, and the output feature map is a fixed size of 8×328×32. Combined with the attention mechanism, the context vector of each time step is calculated using the formula: ; in, is the context vector for each time step, ht is the LSTM hidden state, is the context vector of the previous time step, Wa is the trainable weight matrix.

[0031] The training data includes a synthetic dataset (SynthText) and real-world test images. Data augmentation uses random rotation (±10°), Gaussian noise (σ=0.1σ=0.1), and brightness adjustment (±20%). The loss function is CTC Loss (Connectionist Temporal Classification), the optimizer is Adam, the initial learning rate is 1×10−41×10−4, and it decays by 50% every 10 epochs; Mixed precision training (FP16) is used for model training, and the batch size is set to 32.

[0032] Post-processing and decoding: The sequence recognition output is a character probability matrix, which is decoded by beam search with the beamwidth set to 5; Language model integration: Use the N-gram language model (N=3) to correct the decoding results and improve semantic coherence; The final output is the text sequence with the highest probability, and non-text characters (such as punctuation marks) are filtered out.

[0033] Performance optimization: TensorRT is used to accelerate model inference, and graph optimization and layer fusion are performed on ResNet-50 and LSTM layers; Dynamically quantize the input image (8-bit) to reduce memory usage and improve inference speed.

[0034] Comparative analysis of the present invention and the prior art 1. Regional division accuracy and intelligence Existing technologies: Traditional methods (such as grid segmentation and fixed threshold spacing division) rely on simple text line segmentation or uniform block division, which makes it difficult to distinguish densely arranged questions. For example, if there are no significant gaps or question numbers between test paragraphs, the system may merge multiple questions into one area, resulting in inaccurate retrieval results.

[0035] The present invention improves: Question number detection: Identify question numbers (such as "1.", "2.") through template matching algorithm, and directly divide the area based on the question number.

[0036] Dynamic gap threshold: When there is no question number, the dynamic threshold algorithm T=μ+kσT=μ+kσ (k=1.5k=1.5) is used, combined with Canny edge detection (σ=1.0σ=1.0, high and low threshold ratio 3:1) to accurately extract text line boundaries and distinguish gaps between questions (such as large spaces between questions).

[0037] 2. Innovation of user interaction mechanism: Existing technology: Existing systems usually rely entirely on automatic segmentation, and users cannot intervene in the segmentation results. If the segmentation is wrong, it is necessary to re-shoot or manually crop, which is inefficient.

[0038] Improvements of the present invention: numbered selection and touch selection; the divided areas are displayed with numbers on the terminal device (such as "area 1", "area 2"), and the user selects the target number by touch, triggering the segmentation unit to retain only the selected area. The user takes a photo of a math problem (including multiple sub-questions), and the system divides it into 5 areas. After the user selects the third area, the segmentation unit accurately retains the target sub-question, and the search time is reduced from 10 seconds in the traditional method to 2 seconds.

[0039] 3. Improved robustness of segmentation algorithms Existing technology: Traditional mask segmentation is easily affected by noise, which may lead to mis-segmentation of holes or adhesion areas. For example, when the edges of text lines are blurred, the mask may cover non-target areas.

[0040] Improvements of the present invention: Morphological closing operation: Use 3×3 rectangular kernel for closing operation to fill the holes in the target area and avoid text breaks. Watershed algorithm optimization: Calculate the gradient image (GxGx, GyGy) based on the Sobel operator, combined with the initial markers (Markers) selected by the user, to accurately separate the adhesion area.

[0041] 4. Accuracy and efficiency of OCR models Existing technology: Traditional OCR relies on rule-based feature extraction (such as Tesseract), and has a low recognition rate for blurred text and complex backgrounds (such as handwritten notes on test papers).

[0042] Improvements of the present invention: Deep learning OCR model: Convolutional neural network CNN is used for text detection as the backbone network to extract image features and enhance the robustness against fuzzy text. Bidirectional LSTM: Combines context information to improve sequence recognition accuracy (such as distinguishing between "5" and "S").

[0043] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. The above are only preferred implementation methods of the present invention. It should be pointed out that due to the limitations of textual expression, there are objectively infinite specific structures. For ordinary technicians in this technical field, without departing from the principles of the present invention, they can make several improvements, modifications or changes, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the protection scope of the present invention.

Claims

1. An automatic image subject recognition system, characterized in that: Including terminal equipment, identification and segmentation platform and database; The terminal device is a handheld electronic device, comprising: An image acquisition unit, used to capture and convert images of the real world into digital signals; A display touch unit, used to display images and receive touch selection instructions from users; A communication unit, used for transmitting data with the identification and segmentation platform; The identification and segmentation platform includes: A transmission unit, used for communicating with terminal equipment and database; A pre-processing unit, used to perform white balance, brightness compensation, grayscale and high-resolution sharpening processing on the image; The region recognition module includes a text recognition unit, a gap recognition unit and a division unit, wherein the text recognition unit is used to recognize the title number at the beginning of the text paragraph, the gap recognition unit is used to analyze the paragraph spacing difference, and the division unit divides the title region based on the title number or the spacing feature; The marking unit is used to mark the divided areas and transmit them back to the terminal device; A yes or no selection unit, used to receive the label selected by the user and trigger the segmentation instruction; A segmentation unit, used to perform mask segmentation on non-target areas according to user selection; The information extraction unit uses OCR technology to extract text information from the segmented area; A retrieval unit, used to match the extracted topic information with the database and return the result; The database stores exercise data and is connected to the retrieval unit through the transmission unit.

2. A segmentation method for an automatic image subject recognition system as claimed in claim 1, characterized in that: The following steps are involved: S1. Take a picture of the test question through the terminal device and transmit it to the recognition and segmentation platform; S2. performing preprocessing on the image, including brightness compensation based on histogram equalization, white balance adjustment based on grayscale world assumption, grayscale, and high-resolution sharpening using Laplacian operator; S3. Region Identification: If a title number is detected at the beginning of a text paragraph, the title number is located through a template matching algorithm, and the region is divided using the title number as the boundary; If there is no title number, the text line boundary is extracted through Canny edge detection, and the paragraph spacing is calculated in combination with the dynamic threshold algorithm of the gap recognition unit. If the spacing is greater than the preset threshold, the area is divided using the gap as the dividing line; Gaussian filter standard deviation of the Canny edge detector σ is 1.0, and the ratio of the high threshold to the low threshold is 3:1; S4 will be divided into regional labels and sent back to the terminal device, receiving the user selected label; S5. Segmentation processing: Generates a binary mask based on user selection and applies morphological closing operations to fill holes; The watershed algorithm is used to accurately segment the mask area and retain the target area; S6. Extract the text information of the segmented area through the OCR function, perform similarity matching with the database, and return the matching result.

3. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: The brightness compensation based on histogram equalization in step S2 has an equalization function as follows: Among them, v is the current pixel gray value, ni Grayscale i The number of pixels, N is the total number of pixels, L max is the maximum grayscale level, default L max=255.

4. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: The dynamic threshold algorithm formula in the region identification in step S3 is: Define spacing thresholds T = μ + kσ , in μ is the average spacing, σ is the standard deviation, k is the empirical coefficient, k The value range is 1.2≤ k ≤2.0, default k =1.5, used to distinguish the gaps between questions.

5. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: The watershed algorithm in the segmentation process of step S5 generates a marked area based on the gradient image and separates the adhesion area through the flooding model. The function expression is: Watershed( I grad,Markers) in I Grad is the preprocessed gradient image, and Markers is the initial marker of the user-selected area.

6. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: The morphological closing operation in step S5 uses a 3×3 rectangular kernel, whose structural element is defined as: And the number of closing operation iterations is 1.

7. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: The gradient image of the watershed algorithm in step S5 is calculated by the Sobel operator, and its horizontal and vertical gradient weights are: And the gradient amplitude is calculated as ;in Gx is the horizontal gradient, Gy is the vertical gradient, G is the gradient amplitude.

8. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: The OCR function in step S6 uses a convolutional neural network (CNN) to detect text, combines it with a bidirectional LSTM to recognize sequences, and outputs the text sequence with the highest probability.

9. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: In step S6, the convolutional neural network CNN is used for text detection, and the ResNet-50 backbone network is used: the input image is normalized to 32×256×332×256×3, and multi-scale features are extracted through the convolution layer of ResNet-50; The output feature map generates candidate boxes through RPN. The anchor points of RPN are set to different aspect ratios (0.5, 1, 2) and the scale is [8,16,32][8,16,32] pixels. Non-maximum suppression (NMS): Filter overlapping candidate boxes, suppress redundant detection, set the threshold to 0.7, and ensure that only one optimal candidate box is retained for each text line.

10. The segmentation method of the image subject automatic recognition system according to claim 2, characterized in that: The sequence recognition in step S6 is a bidirectional LSTM and attention mechanism: A bidirectional LSTM network is used, which includes 2 hidden layers, and the number of hidden units in each layer is 256. The LSTM input is the text line image features output by the detection module. The feature extraction is realized through the RoI Align layer, and the output feature map is a fixed size of 8×328×32. Combined with the attention mechanism, the context vector of each time step is calculated using the formula: in, is the context vector for each time step, ht is the LSTM hidden state, is the context vector of the previous time step, Wa is the trainable weight matrix.

Images