A Bimodal Graphic-Text Fusion Object Recognition Method and System

Abstract

The present invention proposes a dual-modal image-text fusion target recognition method and system. The dual-modal image-text fusion target recognition system includes an image recognition module, a text recognition module, and a multi-modal semantic aggregation module. The multi-modal semantic aggregation module is respectively connected to the image recognition module and the text recognition module. The image recognition module uses an image recognition model for target detection. The text recognition module is used to determine whether the recognized text content contains a target phrase. The multi-modal semantic aggregation module outputs a detection result according to the recognition confidence degrees of image recognition and text recognition. In the present invention, an improved YOLOv5 algorithm is used for image recognition, a text localization + text recognition framework is used for text content recognition, the semantic distance similarity and the edit distance similarity are used to calculate the text confidence degree, and the multi-modal information fusion is carried out by means of the text confidence degree, so as to improve the recognition speed on the premise of ensuring the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of graphic and text detection, in particular to a dual-modal graphic and text fusion target recognition method and system. Background Art

[0002] There is a large amount of graphic and text information in cities, such as billboards, social media content, signs, etc. These graphic and text information are not only important objects for urban management, but also important data sources for optimizing resource allocation and promoting sustainable development. Using target detection algorithms to identify them plays an important role in urban management, urban planning and resource allocation.

[0003] For the targets to be detected with both image and text features in the city, simply performing image or text recognition may not achieve the expected goal. Taking the recognition of urban road signs as an example, the text on the road signs may be affected by factors such as occlusion, dirt, or blurring, making it difficult to be accurately recognized. However, the image features such as the color and shape of the road signs are relatively easy to be captured by the image recognition model. Similarly, in the traffic sign detection task, although the image may be difficult to recognize due to problems such as perspective or occlusion, the text recognition model can accurately recognize the traffic instructions on the sign. Since the single-modal model algorithm cannot fully utilize the information of different modalities, it will lead to information asymmetry, which in turn affects the overall performance of the model. Currently, the main method to solve the above problems is to design more complex algorithms, but this will significantly increase the computing resources and time costs. In the scenario of large-scale urban graphic and text information recognition, especially in occasions with high real-time requirements, complex algorithms are difficult to achieve efficient recognition applications. Summary of the Invention

[0004] In order to solve the technical problems in the prior art that the algorithm of the single-modal model is complex, the real-time performance is poor, and it cannot be applied to the large-scale urban graphic and text information recognition scenario, the dual-modal graphic and text fusion target recognition system proposed by the present invention includes an image recognition module, a text recognition module, and a multi-modal semantic aggregation module. The multi-modal semantic aggregation module is respectively connected to the image recognition module and the text recognition module;

[0005] The image recognition module uses an image recognition model to perform target detection;

[0006] The text recognition module is used to judge whether the recognized text content contains a target phrase;

[0007] The multi-modal semantic aggregation module outputs a detection result according to the recognition confidence of image recognition and text recognition.

[0008] Further, the image recognition model is improved based on YOLOv5, including an input end Input, a feature extraction network Backbone, a feature fusion network Neck, and a prediction end Head. The feature fusion network Neck includes a small target detection layer of 160×160 pixels, and a convolutional block attention module is included in the C3 module of the small target detection layer.

[0009] Further, the prediction end Head adopts a decoupled detection head.

[0010] Further, the recognition process of the image recognition model is as follows:

[0011] S11. Process the features of the 80×80 scale feature layer in the feature extraction network Backbone and the upsampled feature layer in the feature fusion network Neck through the C3 module to obtain a deep semantic feature layer containing small target feature information.

[0012] S12. Perform feature splicing and fusion on the deep semantic feature layer and the shallow position feature layer in the feature extraction network Backbone to improve the 160×160 scale fusion feature layer.

[0013] S13. The 160×160 scale fusion feature layer is processed through the C3 module and sent to the prediction end Head.

[0014] S14. Regenerate 12 preset anchor box scales based on the K-means clustering algorithm, including 3 scales for large, medium, small, and extremely small targets respectively.

[0015] Further, the specific working process of the text recognition module is as follows:

[0016] S21. Input the image to be detected into the text detection model DB to determine the text area.

[0017] S22. Input the text area into the text recognition network CRNN to recognize the text content.

[0018] S23. Use a Chinese word segmentation engine to split the recognized text into short phrases, and compare the short phrases with the target phrases to determine whether the recognized text content contains the target phrases.

[0019] Further, during the training process of the text detection model DB, text binarization loss and localization loss are used as the main optimization objectives, and the backpropagation algorithm is used to optimize the overall loss function.

[0020] Further, in S23, if the short phrases after division exist in the Word2Vec corpus, map them to spatial vectors and calculate the semantic distance similarity with the target phrase using cosine similarity. The closer the value of cosine similarity is to 1, the more similar the two words are semantically. If the short phrases after division do not exist in the Word2Vec corpus, calculate the edit distance similarity with the target phrase. Further, the specific working process of the multimodal semantic aggregation module is as follows:

[0021] S31. If the recognition confidence of one modality in the image and text is higher than the preset high threshold, while the confidence of the other modality is lower than the preset low threshold, directly output the single-modal recognition result with higher confidence. If this situation does not occur, proceed to S32;

[0022] S32. Perform weighted summation on the recognition confidences of image recognition and text recognition to calculate the multimodal confidence;

[0023] S33. If the multimodal confidence reaches the preset accuracy, output the target position information and mark it as containing the image-text target. If the multimodal confidence does not reach the preset accuracy, report that the target is not detected.

[0024] Further, in S32, the five-fold cross-validation method is used to optimize the weight parameters.

[0025] The dual-modal image-text fusion recognition method proposed by the present invention is based on the above dual-modal image-text fusion target recognition system. The specific steps of the dual-modal image-text fusion recognition method are as follows:

[0026] S1. The image recognition module uses an image recognition model based on the improved YOLOv5 for target detection;

[0027] S2. The text recognition module determines whether the recognized text content contains the target phrase;

[0028] S3. The multimodal semantic aggregation module outputs the detection result according to the recognition confidences of image recognition and text recognition.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1) Improve the detection module in the YOLOv5 algorithm, add a small target detection layer with a fused convolutional attention module, and regenerate 12 preset anchor box scales based on the K-means clustering algorithm to enhance the small target feature extraction ability and reduce background interference. At the same time, make the anchor boxes more adaptable to the actual size of the targets in the urban scene to improve the robustness of target recognition in complex scenes, and perform image recognition based on the improved YOLOv5 algorithm;

[0031] 2) In text content recognition, a text localization + text recognition framework is designed. Text localization is performed through the text detection model DB, and text confidence is calculated using semantic distance similarity and edit distance similarity for text content recognition. Brief Description of the Drawings

[0032] Figure 1 It is a schematic structural diagram of the dual-modal graphic and text fusion target recognition system of the present invention. Detailed Embodiments

[0033] The technical solution of the present invention will be described in detail below in conjunction with the drawings and specific embodiments.

[0034] As Figure 1 shown, the dual-modal graphic and text fusion target recognition system proposed by the present invention includes an image recognition module, a text recognition module, and a multi-modal semantic aggregation module, and the multi-modal semantic aggregation module is respectively connected to the image recognition module and the text recognition module.

[0035] The image recognition module uses an image recognition model for object detection. The image recognition model is improved based on YOLOv5 and includes an input end (Input), a feature extraction network (Backbone), a feature fusion network (Neck), and a prediction end (Head). Since the downsampling factor of YOLOv5 is relatively large, the resolution of the feature map is relatively low, and the feature information of small targets may be blurred or lost. Therefore, it is relatively difficult to learn the feature information of small targets on deeper feature maps. To solve this problem, a small target detection layer with a size of 160×160 pixels is added to the feature fusion network Neck. It can provide higher-resolution feature information, enabling the model to capture image features and target details more accurately. At the same time, the small target detection layer can perform multi-scale feature extraction and information aggregation on the underlying features, thereby being able to capture more abstract semantic information and improving the model's recognition ability for targets of different scales. However, the small target detection layer introduces more parameters and computational complexity, increasing the complexity of the network. At the same time, making the model focus more on the detection of small targets will sacrifice the detection accuracy of the model for large targets to a certain extent. For this reason, a Convolutional Block Attention Module (CBAM) is introduced into the C3 module of the neural network in the small target detection layer. The importance of each channel and each position in the feature map is dynamically adjusted through the weights generated by channel attention and spatial attention, enabling the network to more effectively capture and utilize important feature information, thereby enhancing the model's object detection and localization ability in complex scenarios. The CBAM module sequentially infers the attention map along two independent dimensions (channel and space), and then multiplies the attention map by the input feature map for adaptive feature refinement. Among them, channel attention allows the model to strengthen the features of specific channels, enabling the model to capture the key semantic features of the target, and spatial attention helps to focus attention at different scales, improving the perception and discrimination ability for different target sizes. The prediction end Head uses a Decoupled Head. The recognition process of the image recognition model is as follows:

[0036] S11. Process the features of the 80×80 scale feature layer in the feature extraction network Backbone and the upsampled feature layer in the feature fusion network Neck through the C3 module to obtain a deep semantic feature layer containing the feature information of small targets.

[0037] S12. Perform feature splicing and fusion on the deep semantic feature layer and the shallow position feature layer in the feature extraction network Backbone to improve the 160×160 scale fusion feature layer and enhance the expression ability of the semantic features and position information of small targets. Let the deep semantic feature layer be F d , and the shallow position feature layer be F s . Then the fusion feature layer is expressed as:

[0038]

[0039] Among them, represents the feature splicing operation.

[0040] S13. The 160×160 scale fusion feature layer is processed by the C3 module and sent to the prediction end Head. The fusion feature layer F fusion After being processed by the C3 module, it is denoted as:

[0041] F processed = C3(F fusion )

[0042] F processed is a deep semantic feature layer containing information about small target features.

[0043] S14. Based on the K-means clustering algorithm, 12 preset anchor box scales are regenerated, including 3 scales for large, medium, small, and extremely small targets each, to solve the situation that the target to be detected is too small while the original anchor box is too large, which is likely to cause the target to be ignored, thereby reducing the missed detection rate of the model. Let the size of the feature map be H×W, and the candidate box at each feature point be B i , and its width and height be (w i , h i ), then the K-means clustering objective function is:

[0044]

[0045] Among them, C is the set of clustering centers, c j is the j-th clustering center, N is the number of candidate boxes, and the finally generated set of anchor box sizes is: {(w1, h1), (w2, h2), …, (w k , h k )}.

[0046] The text recognition module is used to determine whether the recognized text content contains the target phrase. The specific working process of the text recognition module is as follows:

[0047] S21. Input the image to be detected into the text detection model DB to determine the text region. The text detection model DB uses a differentiable method to achieve the binarization of the text image and simultaneously realizes the end-to-end learning process. During the training process of the text detection model DB, the text binarization loss and the localization loss (deformable convolution loss) are used as the main optimization objectives. The text binarization loss is used to measure the difference between the generated text binary image and the labeled text binary image, and the localization loss is used to measure the difference between the generated text position and the labeled text position. The overall loss function can be expressed as the weighted sum of these two loss functions:

[0048] L = αL bin + βL loc

[0049] where α and β are the weight coefficients of the loss function, used to balance the text binarization loss and the localization loss.

[0050] The text binarization loss is:

[0051]

[0052] where is the binary text image generated by the network, yi is the annotated binary text image, and N is the number of samples.

[0053] The localization loss (deformable convolution loss) is:

[0054]

[0055] where is the text position generated by the network, b i is the annotated text position, and N is the number of samples.

[0056] During the training process, the backpropagation algorithm is used to optimize the overall loss function. In the forward propagation process, the output result of the network is calculated through the input data and the current network parameters. In the backpropagation process, first, the gradient of the loss function with respect to the output result is calculated, and then the gradient of the loss function with respect to the network parameters is calculated layer by layer using the chain rule.

[0057] S22. Input the text region into the text recognition network CRNN to recognize the text content. The text recognition network CRNN consists of three parts: feature extraction, sequence modeling, and transcription. In CRNN, CNN is used for feature extraction and RNN is used for sequence modeling.

[0058] CNN extracts local features from the input image data through convolution operations. A series of convolutional kernels are used to convolve the image to capture local feature information such as the texture and edges of the image:

[0059]

[0060] where F is the feature map, σ is the activation function, (x, y) are the coordinates of the feature map, m×n is the size of the convolutional kernel, I(x + i, y + j) is the pixel value of the input image at position (x + i, y + j), K(i, j) is the weight of the convolutional kernel at position (i, j), and b is the bias term.

[0061] The RNN updates the hidden state based on the input features at the current time step and the hidden state at the previous time step. This recursive update process enables the RNN to capture long-term dependencies in the sequence and encode this information into the hidden state, thereby achieving the modeling of sequence information. The hidden state and output of the RNN are expressed as:

[0062] H t =tanh(W ht X t +W hh H t-1 +b h )

[0063] Y t =softmax(W yh H t +b y )

[0064] Where Ht is the hidden state of the RNN at time step t, Xt is the input feature of the input sequence at time step t, Whx is the weight matrix from the input layer to the hidden layer, Whh is the weight matrix from the hidden layer to other hidden layers, bh is the bias vector of the hidden layer, and tanh is the hyperbolic tangent activation function. Yt represents the output of the RNN at time step t, Wyh is the weight matrix from the hidden layer to the output layer, by is the bias vector of the output layer, and softmax is the activation function used for multi-classification tasks.

[0065] By combining CNN and RNN, CRNN can achieve the joint modeling of image and sequence data. Its end-to-end structure enables the model to directly learn from the raw data and make inferences without manually designing features.

[0066] S23. Use the Chinese Word Segmentation Engine (CSWE) to split the recognized text into short phrases, and compare the short phrases with the target phrase to determine whether the recognized text content contains the target phrase. If the short phrases after segmentation exist in the Word2Vec corpus, map them to spatial vectors and calculate the semantic distance similarity with the target phrase using cosine similarity. The closer the value of the cosine similarity is to 1, the more similar the two words are semantically. If the short phrases after segmentation do not exist in the Word2Vec corpus, calculate the edit distance similarity with the target phrase.

[0067] Cosine similarity determines the degree of directional similarity between two vectors in the vector space by measuring the cosine value of the angle between them:

[0068]

[0069] The edit distance algorithm is a method for measuring the similarity between two strings, that is, calculating the minimum number of single-character edits required to convert one string into another, including operations such as inserting, deleting, and replacing characters. The similarity of sentences is inversely proportional to the number of edit operations:

[0070] Sim(s, t) = 1 - LD / Lmax(s, t)

[0071] In the formula, LD represents the edit distance, and Lmax(s, t) represents the maximum length of the two sentences.

[0072] If there are two strings S(s1, s2, …, sm) and T(t1, t2, …, tn), let the matrix LD[m + 1, n + 1]. The algorithm calculates the value of each cell LD(i, j) in the matrix in a loop, then the edit distance LD between strings S and T is calculated by formulas (19), (20)

[22] :

[0073]

[0074] Min = min{LD(i - 1, j) + 1, LD(i, j - 1) + 1, LD(i - 1, j - 1) + f(i, j)

[0075] In the formula, when the ith word of S is not equal to the jth word of T, f(i, j) = 1; otherwise, f(i, j) = 0.

[0076] The multi-modal semantic aggregation module performs a weighted sum of the image confidence and the text similarity, adjusts the weights according to the confidence and similarity of each modality, and reduces the impact of incomplete or unclear information in the text or image by comprehensively using multi-modal information.

[0077] The multi-modal semantic aggregation module outputs the detection result according to the recognition confidence of image recognition and text recognition. The specific working process of the multi-modal semantic aggregation module is as follows:

[0078] S31. If the recognition confidence of one modality in the image and text is higher than the preset high threshold, while the confidence of the other modality is lower than the preset low threshold, directly output the recognition result of the single modality with higher confidence. If this situation does not occur, enter S32.

[0079] S32. Perform a weighted sum of the recognition confidence of image recognition and text recognition to calculate the multi-modal confidence. During the calculation process, adjust the weights according to the recognition confidence of each modality, and reduce the impact of incomplete or unclear information in the text or image by comprehensively using multi-modal information.

[0080] Let the recognition confidence of image recognition be CI, the recognition confidence of text recognition be ST, the weight of the image modality be WI, and the weight of the text modality be WT. The weighted summation formula is as follows:

[0081] F = w I ·C I + w T ·S T

[0082] Where WI and WT satisfy WI + WT = 1.

[0083] To optimize the weight parameters, the present invention introduces a five-fold cross-validation method. The dataset D is divided into five mutually exclusive subsets D1, D2, D3, D4, D5. One of them is selected as the validation set one by one, and the remaining four are used as the training sets. This process is executed five times in a loop to ensure that each subset is verified. The core of this method lies in comparing the performance indicators of each cross-validation under different weight parameters to select the best weight parameter.

[0084] S33. If the multi-modal confidence reaches the preset accuracy, output the target position information and label it as containing the graphic and text target. If the multi-modal confidence does not reach the preset accuracy, report that no target is detected.

[0085] In the urban area, the author independently carried out the shooting work of no-smoking signs in public places, constructed a corresponding dataset, and conducted comparative experiments. The self-created dataset covers the changes of the target to be measured in various scenarios and angles. Before image annotation, the collected images were uniformly scaled to a resolution of 640×640 pixels to ensure that more details and information in the images were retained. There are 9,290 images in the dataset, which are divided into a training set, a test set, and a validation set according to the ratio of 7:2:1. Among them, there are 6,600 images in the training set, 2,000 images in the test set, and 690 images in the validation set. The experiment was carried out using the PyTorch deep learning framework, and the hardware used in the experiment was a GeForce RTX 3060 with 12GB of video memory. Public datasets and Baidu Image Search were used for text recognition training, including LSVT, RCTW-17, MTWI 2018, CASIA-10K, and SROIE, etc. The ratio of the training set to the validation set was set to 8.5:1.5. The Chinese word segmentation engine CSWE was introduced to preprocess the text recognition results to solve the problem of semantic analysis difficulties in long Chinese sentences through word segmentation and semantic unit extraction. At the same time, the RandAugment data augmentation operation was used to improve the image orientation classification effect. When calculating the semantic distance similarity, Word2Vec was used for text vectorization. However, some uncommon expressions of no-smoking signs were not included in the corpus. Therefore, these missing expressions were split, and their average vector values were calculated, and then these new vectors were introduced into the corpus. For example, "Thank you for not smoking" was not included in the corpus. By calculating the average vector of "Thank you", "not", and "smoking", the spatial vector of this expression was generated and added to the corpus. The comparative experiment results show that in both the image and text fields, the dual-modal image-text fusion target recognition system achieved high accuracy rates. The image recognition accuracy rate was 95.58%, and the text recognition accuracy rate was 92%. At the same time, the recognition accuracy under the dual-modal mode reached 97.8%. Compared with the single-modal recognition technology, the accuracy rates were increased by 2.22% and 3.73% respectively.

[0086] The dual-modal image-text fusion recognition method based on the above dual-modal image-text fusion target recognition system specifically includes the following steps:

[0087] S1. The image recognition module uses an image recognition model based on the improved YOLOv5 for target detection;

[0088] S2. The text recognition module determines whether the recognized text content contains the target phrase;

[0089] S3. The multi-modal semantic aggregation module outputs the detection result according to the recognition confidence degrees of the image recognition and the text recognition.

[0090] The above-disclosed is only the preferred embodiment of the present invention, and of course, it cannot be used to limit the scope of the rights of the present invention. It should be pointed out that for those skilled in the art of this technology, any equivalent changes made to the present invention without departing from the design structure and principle of the present invention are regarded as the protection scope of the present invention.

Claims

1. A bimodal graphic-text fusion target recognition system, characterized in that , The dual-modal graphic-text fusion target recognition system includes an image recognition module, a text recognition module, and a multi-modal semantic aggregation module. The multi-modal semantic aggregation module is respectively connected to the image recognition module and the text recognition module; The image recognition module uses an image recognition model for target detection; The text recognition module is used to determine whether the recognized text content contains a target phrase; The multi-modal semantic aggregation module outputs a detection result according to the recognition confidence levels of image recognition and text recognition; The image recognition model is improved based on YOLOv5 and includes an input end Input, a feature extraction network Backbone, a feature fusion network Neck, and a prediction end Head. The feature fusion network Neck includes a small target detection layer of 160×160 pixels, and the C3 module of the small target detection layer includes a convolutional block attention module; The specific working process of the text recognition module is as follows: S21. Input the image to be detected into the text detection model DB to determine the text area; S22. Input the text area into the text recognition network CRNN to recognize the text content; S23. Use a Chinese word segmentation engine to split the recognized text into short phrases, and compare the short phrases with the target phrase to determine whether the recognized text content contains the target phrase; In S23, if the divided short phrase exists in the Word2Vec corpus, map it to a spatial vector, and calculate the semantic distance similarity with the target phrase using cosine similarity. The closer the value of the cosine similarity is to 1, the more similar the two words are semantically. If the divided short phrase does not exist in the Word2Vec corpus, calculate the edit distance similarity with the target phrase; The specific working process of the multi-modal semantic aggregation module is as follows: S31. If the recognition confidence level of one modality in the image and text is higher than a preset high threshold, while the confidence level of the other modality is lower than a preset low threshold, directly output the single-modal recognition result with a higher confidence level. If this situation does not occur, enter S32; S32. Perform weighted summation on the recognition confidence levels of image recognition and text recognition to calculate the multi-modal confidence level; S33. If the multi-modal confidence level reaches the preset accuracy, output the target position information and mark it as containing graphic-text targets. If the multi-modal confidence level does not reach the preset accuracy, report that no target is detected; The prediction end Head uses a decoupled detection head; The recognition process of the image recognition model is as follows: S11. Process the features of the 80×80 scale feature layer in the feature extraction network Backbone and the upsampled feature layer in the feature fusion network Neck through the C3 module to obtain a deep semantic feature layer containing small target feature information; S12. Perform feature splicing and fusion on the deep semantic feature layer and the shallow position feature layer in the feature extraction network Backbone to improve the 160×160 scale fusion feature layer; S13. The 160×160 scale fusion feature layer is processed by the C3 module and sent to the prediction end Head; S14. Regenerate 12 preset anchor box scales based on the K-means clustering algorithm, including 3 scales for large, medium, small, and extremely small targets respectively.

2. The dual-modal graphic-text fusion target recognition system according to claim 1, wherein During the training process of the text detection model DB, the text binarization loss and the localization loss are used as the main optimization objectives, and the backpropagation algorithm is used to optimize the overall loss function.

3. The dual-modal graphic-text fusion object recognition system according to claim 1, wherein In S32, the five-fold cross-validation method is used to optimize the weight parameters.

4. A dual-modal graphic-text fusion recognition method for the dual-modal graphic-text fusion target recognition system according to any one of claims 1-3, characterized in that , The dual-modal image-text fusion recognition method specifically includes the following steps: S1. The image recognition module uses an image recognition model based on the improved YOLOv5 for target detection; S2. The text recognition module determines whether the recognized text content contains the target phrase; S3. The multi-modal semantic aggregation module outputs the detection result according to the recognition confidence of the image recognition and the text recognition.

Images