An image subject automatic recognition system and segmentation method

Abstract

The present application relates to the technical field of image data processing, and especially to an image subject automatic identification system and a segmentation method, comprising a terminal device, an identification and segmentation platform and a database; the terminal device is a handheld electronic device, which is a physical device for connecting a user with the identification and segmentation platform in a communication network, and comprises an image acquisition unit, a display touch unit and a communication unit; the segmentation method of the image subject automatic identification system comprises the following steps: S1. capturing a test image through the terminal device and transmitting the test image to the identification and segmentation platform; S2. performing pre-processing on the image; S3. region identification; S4. returning the region labels after division to the terminal device and receiving the labels selected by a user; S5. segmentation processing; and S6. extracting text information of the segmented region through an OCR function and performing similarity matching with a database to return a matching result. The present application has high-precision region division, algorithm robustness and fast retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to an automatic image subject recognition system and a segmentation method. Background Art

[0002] With the advancement of network technology, online question search has become popular. In general, the technology is to collect test information through terminal devices (such as mobile phones, tablets, etc.) and upload it to the cloud server. The server analyzes and processes the input information, and after comparing it with the massive exercise data in the database, the highly matched results are returned to the terminal to complete the question search. The collection of test information generally requires photography, so an automatic image subject recognition system is needed to identify the test information in the picture and segment the test information.

[0003] Existing automatic image subject recognition systems usually recognize text and automatically segment the image area through a raster method to separate the test question information. However, since a large amount of test question text is generally densely arranged together, this recognition and segmentation method is difficult to accurately identify which part of the question the student is searching for in the image, and thus cannot specifically segment the required search part. Therefore, based on the above problems, an automatic image subject recognition system and segmentation method are proposed. Summary of the Invention

[0004] The purpose of the present invention is to provide an automatic image subject recognition system and segmentation method to solve the problem that the existing automatic image subject recognition system is difficult to accurately identify which part of the question the user wants to search for from a large amount of text in the image, and is unable to specifically segment the required search part.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides an automatic image subject recognition system, comprising a terminal device, a recognition and segmentation platform, and a database;

[0007] The terminal device is a handheld electronic device, comprising:

[0008] An image acquisition unit, used to capture and convert images of the real world into digital signals;

[0009] A display touch unit, used to display images and receive touch selection instructions from the user;

[0010] A communication unit, used for transmitting data with the identification and segmentation platform;

[0011] The identification and segmentation platform includes:

[0012] A transmission unit, used for communicating with terminal devices and databases;

[0013] Pre-processing unit, used to perform white balance, brightness compensation, grayscale and high-resolution sharpening on the image;

[0014] A region recognition module includes a text recognition unit, a gap recognition unit, and a division unit. The text recognition unit is used to recognize the title number at the beginning of a text paragraph, the gap recognition unit is used to analyze the difference in paragraph spacing, and the division unit divides the title region based on the title number or spacing characteristics.

[0015] The marking unit is used to mark the divided areas and transmit them back to the terminal device;

[0016] A yes / no selection unit, used to receive the label selected by the user and trigger the split instruction;

[0017] A segmentation unit, configured to perform mask segmentation on non-target areas according to user selection;

[0018] Information extraction unit, which uses OCR technology to extract text information in the segmented area;

[0019] A retrieval unit, used to match the extracted topic information with the database and return the results;

[0020] The database stores exercise data and is connected to the retrieval unit through the transmission unit.

[0021] In a second aspect, the present invention further provides a segmentation method based on an automatic image subject recognition system, comprising the following steps:

[0022] S1. Capture the test image using a terminal device and transmit it to the recognition and segmentation platform;

[0023] S2. Perform image preprocessing, including brightness compensation based on histogram equalization, white balance adjustment based on the grayscale world assumption, grayscaling, and high-resolution sharpening using the Laplacian operator.

[0024] S3. Region Identification:

[0025] If a title number is detected at the beginning of a text paragraph, the title number is located using a template matching algorithm, and the region is divided using the title number as the boundary.

[0026] If there is no title number, the text line boundary is extracted through Canny edge detection, and the paragraph spacing is calculated in combination with the dynamic threshold algorithm of the gap recognition unit. If the spacing is greater than the preset threshold, the region is divided based on the gap.

[0027] Gaussian filter standard deviation of the Canny edge detector σ is 1.0, and the ratio of the high threshold to the low threshold is 3:1;

[0028] S4. Return the divided region label to the terminal device, and receive the label selected by the user;

[0029] S5. Segmentation processing:

[0030] Generate a binary mask based on user selection, and fill holes by applying morphological closing operation;

[0031] Use the watershed algorithm to accurately segment the mask region and retain the target region;

[0032] S6. Extract the text information of the segmented region through the OCR function, and perform similarity matching with the database to return the matching result.

[0033] Further, the brightness compensation based on histogram equalization in step S2 has an equalization function: ;

[0034] Where v is the current pixel gray value, ni is the number of pixels at the gray level i , N is the total number of pixels, L max is the maximum gray level, and by default L max=255.

[0035] Further, the dynamic threshold algorithm formula in the region identification of step S3 is:

[0036] Define the distance threshold T = μ + kσ ,

[0037] Where μ is the average distance, σ is the standard deviation, k is the empirical coefficient, k The value range of k is 1.2≤ k ≤2.0, and by default =1.5, used to distinguish the gap between questions.

[0038] Further, the watershed algorithm in the segmentation processing of step S5 generates a labeled region based on the gradient image, separates the adherent region by the flooding model, and the function expression is:

[0039] I Watershed( grad, Markers)

[0040] Where I grad is the preprocessed gradient image, and Markers is the initial label of the user-selected region.

[0041] Furthermore, the morphological closing operation in step S5 uses a 3×3 rectangular kernel, whose structural element is defined as: ;

[0042] And the number of closing operation iterations is 1.

[0043] Furthermore, the gradient image of the watershed algorithm in step S5 is calculated by the Sobel operator, and its horizontal and vertical gradient weights are respectively: ;

[0044] And the gradient amplitude is calculated as ;in Gx is the horizontal gradient, Gy is the vertical gradient, G is the gradient amplitude.

[0045] Furthermore, the OCR function in step S6 uses a convolutional neural network (CNN) for text detection and combines it with a bidirectional LSTM for sequence recognition to output the text sequence with the highest probability.

[0046] Furthermore, in step S6, a convolutional neural network (CNN) is used for text detection, using a ResNet-50 backbone network: the input image is normalized to 32×256×332×256×3, and multi-scale features are extracted through the convolutional layer of ResNet-50;

[0047] The output feature map is passed through RPN to generate candidate boxes. The anchor points of RPN are set to different aspect ratios (0.5, 1, 2) and the scale is [8, 16, 32][8, 16, 32] pixels.

[0048] Non-maximum suppression (NMS): Filters overlapping candidate boxes, suppresses redundant detections, and sets the threshold to 0.7 to ensure that only one optimal candidate box is retained for each text line.

[0049] Furthermore, the sequence recognition in step S6 is a bidirectional LSTM and attention mechanism:

[0050] A bidirectional LSTM network is used, consisting of two hidden layers with 256 hidden units in each layer. The LSTM input is the text line image features output by the detection module. Feature extraction is performed through the RoI Align layer, which outputs a feature map with a fixed size of 8×328×32.

[0051] Combined with the attention mechanism, the context vector of each time step is calculated using the formula: ;

[0052] wherein, is a context vector for each time step, ht is an LSTM hidden state, is a context vector for the previous time step, Wa is a trainable weight matrix.

[0053] The present application has the following advantages:

[0054] High-precision region division: combined with topic number detection and dynamic gap threshold, solves the problem of dense text segmentation.

[0055] User interaction optimization: label box selection and touch selection mechanism, improve the efficiency of target area positioning.

[0056] Algorithm robustness: combination of watershed algorithm and morphological closing operation, to avoid over-segmentation or under-segmentation.

[0057] Fast search: efficient matching of OCR model and database, realizing second-level search response. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a system structure diagram of the present application;

[0059] Figure 2 is an image preprocessing core code diagram of the present application;

[0060] Figure 3 is a region division and segmentation core code diagram of the present application;

[0061] Figure 4 is an OCR text extraction core code diagram of the present application;

[0062] Figure 5 is a traditional raster segmentation diagram;

[0063] Figure 6 is a raster segmentation comparison diagram of the present application. DETAILED DESCRIPTION

[0064] Please refer to Figure 1 , the present application provides a kind of technical scheme:

[0065] Embodiment 1, as Figure 1 shown, an image subject automatic identification system includes terminal device, identification segmentation platform and database;

[0066] The terminal device is handheld electronic device, including:

[0067] Image acquisition unit, for capturing and converting the image of real world into digital signal;

[0068] a display touch unit for displaying images and receiving touch selection instructions of a user;

[0069] a communication unit for data transmission with the recognition and segmentation platform;

[0070] the recognition and segmentation platform comprises:

[0071] a transmission unit for communication with terminal equipment and a database;

[0072] a preprocessing unit for white balance, brightness compensation, grayscale and high-resolution sharpening processing of images;

[0073] a region recognition module, including a text recognition unit, a gap recognition unit and a division unit, the text recognition unit is used for recognizing the title number at the beginning of the text paragraph, the gap recognition unit is used for analyzing the difference in paragraph spacing, and the division unit is based on the title number or spacing feature to divide the title region;

[0074] a marking unit for marking and framing the divided region and returning to the terminal equipment;

[0075] a yes / no selection unit for receiving user-selected labels and triggering segmentation instructions;

[0076] a segmentation unit for masking and segmenting non-target regions according to user selection;

[0077] an information extraction unit for extracting text information of the segmented region using OCR technology;

[0078] a retrieval unit for matching the extracted title information with the database and returning the result;

[0079] the database stores the exercise materials and is connected with the retrieval unit through the transmission unit.

[0080] Embodiment 2, as shown in a segmentation method based on an image main body automatic recognition system, comprising the following steps: Figure 2-Figure 6

[0081] S1. Taking a test image through a terminal equipment and transmitting it to a recognition and segmentation platform;

[0082] S2. Preprocessing the image, including brightness compensation based on histogram equalization, white balance adjustment based on gray world assumption, grayscale and high-resolution sharpening using Laplace operator;

[0083] wherein the user takes a test paper image, the system performs brightness compensation and edge enhancement, and the brightness compensation based on histogram equalization has an equalization function as follows: ; ​

[0084] wherein v is the current pixel gray value, ni is the gray level i of the pixel number, N is the total pixel number, L max is the maximum gray level, by default L max=255.

[0085] The core code is shown in Figure 2 :

[0086] S3. Region identification:

[0087] If the title number is detected at the beginning of the text paragraph, the title number is located by a template matching algorithm, and the region is divided with the title number as the boundary;

[0088] If there is no title number, the text line boundary is extracted by Canny edge detection, and the paragraph spacing is calculated by combining the dynamic threshold algorithm of the gap recognition unit. If the spacing is greater than the preset threshold, the gap is taken as the division line to divide the region;

[0089] The standard deviation of the Gaussian filter of the Canny edge detection is 1.0, and the ratio of the high threshold value to the low threshold value is 3:1. σ

[0090] The dynamic threshold algorithm formula in the step S3 region identification is:

[0091] The spacing threshold is defined as T = μ + kσ ,

[0092] wherein μ is the average spacing, σ is the standard deviation, k is an empirical coefficient, k the value range of k is 1.2≤ k ≤2.0, by default =1.5, for distinguishing the gap between titles.

[0093] As shown in Figure 5 , the traditional method merges 3 questions into 1 region; as shown in Figure 6 : the present application correctly divides into 3 regions.

[0094] The core code is shown in Figure 3 :

[0095] S4. The region label after division is returned to the terminal device, and the label selected by the user is received;

[0096] S5. Segmentation processing:

[0097] Generate a binary mask based on user selection and apply morphological closing operations to fill holes;

[0098] The watershed algorithm is used to accurately segment the mask area and retain the target area;

[0099] The watershed algorithm in the segmentation process of step S5 generates a marked area based on the gradient image and separates the adhesion area through the submerged model. The function expression is:

[0100] Watershed I grad,Markers)

[0101] in I grad is the preprocessed gradient image, and Markers is the initial marker of the user-selected area.

[0102] The morphological closing operation uses a 3×3 rectangular kernel, whose structural element is defined as: ;

[0103] And the number of closing operation iterations is 1.

[0104] The gradient image of the watershed algorithm is calculated by the Sobel operator, and its horizontal and vertical gradient weights are: ;

[0105] And the gradient amplitude is calculated as ;in Gx is the horizontal gradient, Gy is the vertical gradient, G is the gradient amplitude.

[0106] S6. Use the OCR function to extract the text information of the segmented area, perform similarity matching with the database, and return the matching results. The core code is as follows: Figure 4 shown.

[0107] In step S6, a convolutional neural network (CNN) is used for text detection, using a ResNet-50 backbone network: the input image is normalized to 32×256×332×256×3, and multi-scale features are extracted through the convolutional layer of ResNet-50;

[0108] The output feature map is passed through RPN to generate candidate boxes. The anchor points of RPN are set to different aspect ratios (0.5, 1, 2) and the scale is [8, 16, 32][8, 16, 32] pixels.

[0109] Non-maximum suppression (NMS): Filters overlapping candidate boxes, suppresses redundant detections, and sets the threshold to 0.7 to ensure that only one optimal candidate box is retained for each text line.

[0110] The sequence recognition in step S6 is a bidirectional LSTM and attention mechanism. The core code is as follows Figure 4 As shown:

[0111] A bidirectional LSTM network is used, consisting of two hidden layers with 256 hidden units in each layer. The LSTM input is the text line image features output by the detection module. Feature extraction is performed through the RoI Align layer, which outputs a feature map with a fixed size of 8×328×32.

[0112] Combined with the attention mechanism, the context vector of each time step is calculated using the formula: ;

[0113] in, is the context vector for each time step, ht is the LSTM hidden state, is the context vector of the previous time step, Wa is the trainable weight matrix.

[0114] The training data includes a synthetic dataset (SynthText) and real-world test images. Data augmentation uses random rotation (±10°), Gaussian noise (σ=0.1σ=0.1), and brightness adjustment (±20%).

[0115] The loss function is CTC Loss (Connectionist Temporal Classification), the optimizer is Adam, the initial learning rate is 1×10−41×10−4, and it decays by 50% every 10 epochs;

[0116] The model is trained using mixed precision training (FP16) and the batch size is set to 32.

[0117] Post-processing and decoding: The sequence recognition output is a character probability matrix, which is decoded by beam search with a beamwidth of 5.

[0118] Language model integration: Use the N-gram language model (N=3) to correct the decoding results and improve semantic coherence;

[0119] The final output is the text sequence with the highest probability, and non-text characters (such as punctuation marks) are filtered out.

[0120] Performance optimization: TensorRT is used to accelerate model inference, and graph optimization and layer fusion are performed on ResNet-50 and LSTM layers.

[0121] Dynamically quantize the input image (8-bit) to reduce memory usage and improve inference speed.

[0122] Comparative analysis of the present invention and the prior art

[0123] 1. Regional division accuracy and intelligence

[0124] Existing Technologies: Traditional methods (such as grid segmentation and fixed-threshold spacing) rely on simple text line segmentation or uniform block division, making it difficult to distinguish densely packed questions. For example, if there are no significant gaps or question numbers between paragraphs, the system may merge multiple questions into a single area, resulting in inaccurate search results.

[0125] The present invention improves:

[0126] Question number detection: Identify question numbers (such as "1.", "2.") through template matching algorithms and directly divide areas based on the question numbers.

[0127] Dynamic gap threshold: When there are no question numbers, the dynamic threshold algorithm T = μ + kσ (k = 1.5k = 1.5) is used, combined with Canny edge detection (σ = 1.0σ = 1.0, high and low threshold ratio 3:1) to accurately extract text line boundaries and distinguish gaps between questions (such as large spaces between questions).

[0128] 2. Innovation in user interaction mechanism:

[0129] Existing technologies: Existing systems typically rely entirely on automatic segmentation, leaving users unable to manipulate the results. Segmentation errors require reshooting or manual cropping, which is inefficient.

[0130] This invention improves on the previous approach by enabling numbered selection and touch-screen selection. The divided areas are displayed on the terminal device with labels (e.g., "Area 1," "Area 2"). The user touches the target label, triggering the segmentation unit to retain only the selected area. For example, a user takes a photo of a math problem (containing multiple subproblems), and the system divides it into five areas. After the user selects the third area, the segmentation unit accurately retains the target subproblem, reducing search time from 10 seconds with traditional methods to 2 seconds.

[0131] 3. Improved robustness of segmentation algorithms

[0132] Existing technologies: Traditional mask segmentation is susceptible to noise interference, resulting in mis-segmentation of holes or adhesion areas. For example, when the edges of text lines are blurred, the mask may cover non-target areas.

[0133] Improvements in this invention: Morphological closing operation: A 3×3 rectangular kernel is used for closing operations to fill holes in the target area and avoid text breaks. Watershed algorithm optimization: The gradient image (GxGx, GyGy) is calculated based on the Sobel operator, combined with the initial markers selected by the user, to accurately separate the adhesion areas.

[0134] 4. Accuracy and efficiency of OCR models

[0135] Existing technology: Traditional OCR relies on rule-based feature extraction (such as Tesseract), which has a low recognition rate for blurred text and complex backgrounds (such as handwritten notes on test papers).

[0136] Improvements in this invention: Deep learning OCR model: Convolutional neural networks (CNNs) are used as the backbone for text detection to extract image features, enhancing robustness against ambiguous text. Bidirectional LSTMs incorporate contextual information to improve sequence recognition accuracy (e.g., distinguishing between "5" and "S").

[0137] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method of the present invention and its core ideas. The above are only preferred implementation methods of the present invention. It should be pointed out that due to the limitations of textual expression, there are objectively infinite specific structures. For ordinary technicians in this technical field, without departing from the principles of the present invention, they can make several improvements, modifications or changes, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the scope of protection of the present invention.

Claims

1. A segmentation method for an automatic image subject recognition system, comprising a terminal device, a recognition and segmentation platform, and a database; the terminal device is a handheld electronic device, characterized in that: include: An image acquisition unit, used to capture and convert images of the real world into digital signals; A display touch unit, used to display images and receive touch selection instructions from the user; A communication unit, used for transmitting data with the identification and segmentation platform; The identification and segmentation platform includes: A transmission unit, used for communicating with terminal devices and databases; Pre-processing unit, used to perform white balance, brightness compensation, grayscale and high-resolution sharpening on the image; A region recognition module includes a text recognition unit, a gap recognition unit, and a division unit. The text recognition unit is used to recognize the title number at the beginning of a text paragraph, the gap recognition unit is used to analyze the difference in paragraph spacing, and the division unit divides the title region based on the title number or spacing characteristics. The marking unit is used to mark the divided areas and transmit them back to the terminal device; A yes / no selection unit, used to receive the label selected by the user and trigger the split instruction; The watershed algorithm is used to accurately segment the mask area and retain the target area; Information extraction unit, which uses OCR technology to extract text information in the segmented area; A retrieval unit, used to match the extracted topic information with the database and return the results; The database stores exercise data and is connected to the retrieval unit via the transmission unit; The method comprises the following steps: S1. Capture the test image through the terminal device and transmit it to the recognition and segmentation platform; S2. performing image preprocessing, including brightness compensation based on histogram equalization, white balance adjustment based on the grayscale world assumption, grayscaling, and high-resolution sharpening using the Laplacian operator; S3. Region Identification: If a title number is detected at the beginning of a text paragraph, the title number is located using a template matching algorithm, and the region is divided using the title number as the boundary. If there is no title number, the text line boundary is extracted through Canny edge detection, and the paragraph spacing is calculated in combination with the dynamic threshold algorithm of the gap recognition unit. If the spacing is greater than the preset threshold, the region is divided based on the gap. The Gaussian filter standard deviation σ of the Canny edge detection is 1.0, and the ratio of the high threshold to the low threshold is 3:1; S4 will be divided into regional labels and sent back to the terminal device, receiving the user's selected label; S5. Segmentation processing: Generate a binary mask based on user selection and apply morphological closing operations to fill holes; The watershed algorithm is used to accurately segment the mask area and retain the target area; S6. Extract the text information of the segmented area through the OCR function, perform similarity matching with the database, and return the matching result.

2. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: The brightness compensation based on histogram equalization in step S2 has an equalization function: Among them, v is the gray value of the current pixel, ni is the number of pixels of gray level i, N is the total number of pixels, L max The maximum gray level, the default is L max =255.

3. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: The dynamic threshold algorithm formula in the region identification in step S3 is: Define the spacing threshold T = μ + kσ, Where μ is the average spacing, σ is the standard deviation, and k is the empirical coefficient. The value range of k is 1.2≤k≤2.0, and the default k=1.5, which is used to distinguish the gaps between questions.

4. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: The watershed algorithm in the segmentation process of step S5 generates a marked area based on the gradient image and separates the adhesion area through the submerged model. The function expression is: Watershed(Igrad,Markers) Where Igrad is the preprocessed gradient image, and Markers is the initial marker of the user-selected area.

5. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: The morphological closing operation in step S5 uses a 3×3 rectangular kernel, whose structural element is defined as: And the number of closing operation iterations is 1.

6. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: The gradient image of the watershed algorithm in step S5 is calculated using the Sobel operator, and its horizontal and vertical gradient weights are: And the gradient amplitude is calculated as Where Gx is the horizontal gradient, Gy is the vertical gradient, and G is the gradient amplitude.

7. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: The OCR function in step S6 uses a convolutional neural network (CNN) for text detection and a bidirectional LSTM for sequence recognition to output the text sequence with the highest probability.

8. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: In step S6, a convolutional neural network (CNN) is used for text detection, using a ResNet-50 backbone network: the input image is normalized to 32×256×332×256×3, and multi-scale features are extracted through the convolutional layer of ResNet-50; The output feature map is passed through RPN to generate candidate boxes. The anchor points of RPN are set to different aspect ratios (0.5, 1, 2) and the scale is [8, 16, 32][8, 16, 32] pixels. Non-maximum suppression (NMS): Filters overlapping candidate boxes, suppresses redundant detections, and sets the threshold to 0.7 to ensure that only one optimal candidate box is retained for each text line.

9. The segmentation method of the image subject automatic recognition system according to claim 1, characterized in that: The sequence recognition in step S6 is a bidirectional LSTM and attention mechanism: A bidirectional LSTM network is used, consisting of two hidden layers with 256 hidden units in each layer. The LSTM input is the text line image features output by the detection module. Feature extraction is performed through the RoI Align layer, which outputs a feature map with a fixed size of 8×328×32. Combined with the attention mechanism, the context vector of each time step is calculated using the formula: a t =Softmax(W a ·[h t ;c t-1 ]) Among them, α t is the context vector for each time step, ht is the LSTM hidden state, ct-1 is the context vector of the previous time step, and Wa is the trainable weight matrix.

Images