Bimodal image-text fusion target identification method and system

Abstract

The invention provides a bimodal image-text fusion target recognition method and system.The bimodal image-text fusion target recognition system comprises an image recognition module, a character recognition module and a multi-modal semantic aggregation module, and the multi-modal semantic aggregation module is connected with the image recognition module and the character recognition module. The image recognition module adopts an image recognition model to carry out target detection, the character recognition module is used for judging whether recognized character content contains a target phrase or not, and the multi-modal semantic aggregation module outputs a detection result according to recognition confidence of image recognition and character recognition. According to the method, image recognition is carried out by adopting an improved YOLOv5 algorithm, character content recognition is carried out by adopting a character positioning and character recognition framework, the text confidence coefficient is calculated by adopting semantic distance similarity and editing distance similarity, multi-modal information fusion is carried out by virtue of the text confidence coefficient, and the recognition speed is improved on the premise of ensuring the recognition precision.

Description

Technical Field

[0001] The present invention relates to the technical field of image and text detection, and in particular to a dual-modal image and text fusion target recognition method and system. Background Art

[0002] There is a large amount of graphic information in the city, such as billboards, social media content, signboards, etc. These graphic information are not only important objects of urban management, but also important data sources for optimizing resource allocation and promoting sustainable development. Using target detection algorithms to identify them plays an important role in urban management, urban planning and resource allocation.

[0003] For the targets to be detected in the city with dual features of images and text, simply performing image or text recognition may not achieve the expected goal. Taking urban road sign recognition as an example, the text on the road sign may be affected by factors such as occlusion, stain or blur, making it difficult to accurately recognize, but the image features such as the color and shape of the road sign are relatively easy to be captured by the image recognition model. Similarly, in the task of traffic sign detection, although the image may be difficult to recognize due to problems such as perspective or occlusion, the text recognition model can accurately recognize the traffic instructions on the sign. Since the single-modal model algorithm cannot make full use of information from different modalities, it will lead to information asymmetry, which will in turn affect the overall performance of the model. The current solution to the above problems is mainly to design more complex algorithms, but this will significantly increase computing resources and time costs. In large-scale urban graphic information recognition scenarios, especially in situations with high real-time requirements, complex algorithms are difficult to achieve efficient recognition applications. Summary of the invention

[0004] In order to solve the technical problems that the single modal model in the prior art has a complex algorithm, poor real-time performance, and cannot be applied to large-scale urban image and text information recognition scenarios, the dual-modal image and text fusion target recognition system proposed in the present invention includes an image recognition module, a text recognition module and a multi-modal semantic aggregation module, and the multi-modal semantic aggregation module is respectively connected to the image recognition module and the text recognition module;

[0005] The image recognition module uses an image recognition model to perform target detection;

[0006] The text recognition module is used to determine whether the recognized text content contains a target phrase;

[0007] The multimodal semantic aggregation module outputs detection results according to the recognition confidence of image recognition and text recognition.

[0008] Furthermore, the image recognition model is improved based on YOLOv5, including an input end Input, a feature extraction network Backbone, a feature fusion network Neck and a prediction end Head, the feature fusion network Neck includes a small target detection layer of 160×160 pixels, and the C3 module of the small target detection layer includes a convolutional block attention module.

[0009] Furthermore, the prediction end Head adopts a decoupled detection head.

[0010] Furthermore, the recognition process of the image recognition model is as follows:

[0011] S11, the features of the 80×80 scale feature layer in the feature extraction network Backbone and the up-sampled feature layer in the feature fusion network Neck are processed by the C3 module to obtain a deep semantic feature layer containing the feature information of the small target;

[0012] S12, concatenate and fuse the deep semantic feature layer with the shallow position feature layer in the feature extraction network Backbone to improve the 160×160 scale fusion feature layer;

[0013] S13, 160×160 scale fusion feature layer is processed by C3 module and sent to the prediction end Head;

[0014] S14. Regenerate 12 preset anchor box scales based on the K-means clustering algorithm, including 3 scales each for large, medium, small and extremely small targets.

[0015] Furthermore, the specific working process of the text recognition module is as follows:

[0016] S21, input the image to be detected into the text detection model DB to determine the text area;

[0017] S22, inputting the text area into the text recognition network CRNN to recognize the text content;

[0018] S23. Use a Chinese word segmentation engine to split the recognized text into short words, and compare the short words with the target phrase to determine whether the recognized text content contains the target phrase.

[0019] Furthermore, during the training process of the text detection model DB, text binarization loss and positioning loss are used as the main optimization targets, and the back propagation algorithm is used to optimize the overall loss function.

[0020] Furthermore, in S23, if the short words after the division exist in the Word2Vec corpus, they are mapped into space vectors, and the semantic distance similarity is calculated with the target phrase using cosine similarity. The closer the cosine similarity value is to 1, the more similar the two words are in semantics. If the short words after the division do not exist in the Word2Vec corpus, the edit distance similarity is calculated with the target phrase. Furthermore, the specific working process of the multimodal semantic aggregation module is as follows:

[0021] S31, if the recognition confidence of one modality among the image and the text is higher than the preset high threshold, and the confidence of the other modality is lower than the preset low threshold, directly output the recognition result of the single modality with higher confidence; if this situation does not occur, enter S32;

[0022] S32, performing weighted summation on the recognition confidence of the image recognition and the text recognition to calculate a multimodal confidence;

[0023] S33. If the multimodal confidence reaches the preset accuracy, the target location information is output and it is marked as containing a graphic target. If the multimodal confidence does not reach the preset accuracy, it will be reported that the target is not detected.

[0024] Furthermore, in S32, a five-fold cross validation method is used to optimize the weight parameters.

[0025] The dual-modal image-text fusion recognition method proposed in the present invention is based on the dual-modal image-text fusion target recognition system, and the dual-modal image-text fusion recognition method specifically includes the following steps:

[0026] S1, the image recognition module uses an improved YOLOv5-based image recognition model for target detection;

[0027] S2, the text recognition module determines whether the recognized text content contains the target phrase;

[0028] S3, the multimodal semantic aggregation module outputs the detection results according to the recognition confidence of image recognition and text recognition.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1) Improve the detection module in the YOLOv5 algorithm, add a small target detection layer that integrates the convolutional attention module, and regenerate 12 preset anchor box scales based on the K-means clustering algorithm to enhance the small target feature extraction capability and reduce background interference. At the same time, make the anchor box more suitable for the actual size of the target in the urban scene to improve the robustness of target recognition in complex scenes, and perform image recognition based on the improved YOLOv5 algorithm;

[0031] 2) In text content recognition, a text positioning + text recognition framework is designed, text positioning is performed through the text detection model DB, and the text confidence is calculated using semantic distance similarity and edit distance similarity to perform text content recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a structural schematic diagram of the dual-modal image-text fusion target recognition system of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0034] like Figure 1 As shown, the dual-modal image-text fusion target recognition system proposed in the present invention includes an image recognition module, a text recognition module and a multi-modal semantic aggregation module, and the multi-modal semantic aggregation module is respectively connected to the image recognition module and the text recognition module.

[0035] The image recognition module uses an image recognition model for target detection. The image recognition model is improved based on YOLOv5, including the input end Input, the feature extraction network Backbone, the feature fusion network Neck and the prediction end Head. Since the YOLOv5 downsampling multiple is relatively large, the resolution of the feature map is relatively low, and the feature information of small targets may be blurred or lost. Therefore, it is relatively difficult to learn the feature information of small targets on deeper feature maps. In order to solve this problem, a 160×160 pixel small target detection layer is added to the feature fusion network Neck, which can provide higher resolution feature information, so that the model can more accurately capture image features and target details. At the same time, the small target detection layer can perform multi-scale feature extraction and information aggregation on the underlying features, so as to capture more abstract semantic information and improve the model's recognition ability for targets of different scales. However, the small target detection layer introduces more parameters and calculations, increases the complexity of the network, and makes the model pay more attention to the detection of small targets, which will sacrifice the model's detection accuracy for large targets to a certain extent. For this reason, the convolutional block attention module is introduced in the neural network C3 module of the small target detection layer. Module, CBAM), dynamically adjusts the importance of each channel and each position in the feature map through the weights generated by channel attention and spatial attention, so that the network can more effectively capture and utilize important feature information, thereby improving the model's target detection and positioning capabilities in complex scenes. The CBAM module infers the attention map along two independent dimensions (channel and space) in sequence, and then multiplies the attention map with the input feature map for adaptive feature refinement. Among them, channel attention allows the model to strengthen the features of specific channels, so that the model can capture the key semantic features of the target, and spatial attention helps to focus attention at different scales and improve the perception and distinction of different target sizes. The prediction end Head uses a decoupled detection head (Decoupled Head). The recognition process of the image recognition model is as follows:

[0036] S11. The features of the 80×80 scale feature layer in the feature extraction network Backbone and the upsampling feature layer in the feature fusion network Neck are processed by the C3 module to obtain a deep semantic feature layer containing feature information about small targets.

[0037] S12, concatenate and fuse the deep semantic feature layer with the shallow position feature layer in the feature extraction network Backbone to improve the 160×160 scale fusion feature layer and enhance the ability to express the semantic features and position information of small targets. Let the deep semantic feature layer be F d , the shallow position feature layer is F s , then the fusion feature layer is expressed as:

[0038]

[0039] in, Represents a feature concatenation operation.

[0040] S13, 160×160 scale fusion feature layer is processed by C3 module and sent to the prediction end Head. fusion After being processed by the C3 module, it is recorded as:

[0041] F processed =C3(F fusion )

[0042] F processed It is a deep semantic feature layer that contains information about small object features.

[0043] S14. Based on the K-means clustering algorithm, 12 preset anchor box scales are regenerated, including 3 scales for large, medium, small and very small targets, to solve the problem that the target to be measured is too small and the original anchor box is too large, which may cause the target to be ignored, thereby reducing the missed detection rate of the model. Assume that the size of the feature map is H×W, and the candidate box on each feature point is B i , its width and height are (w i ,h i ), then the K-means clustering objective function is:

[0044]

[0045] Among them, C is the set of cluster centers, c j is the jth cluster center, N is the number of candidate boxes, and the final generated anchor box size set is: {(w1,h1),(w2,h2),…,(w k ,h k )}.

[0046] The text recognition module is used to determine whether the recognized text content contains the target phrase. The specific working process of the text recognition module is as follows:

[0047] S21. Input the image to be detected into the text detection model DB to determine the text area. The text detection model DB uses a differentiable method to realize the binarization of the text image and realizes an end-to-end learning process. In the training process of the text detection model DB, the text binarization loss and the positioning loss (variable convolution loss) are used as the main optimization targets. The text binarization loss is used to measure the difference between the generated text binary image and the annotated text binary image. The positioning loss is used to measure the difference between the generated text position and the annotated text position. The overall loss function can be expressed as the weighted sum of these two loss functions:

[0048] L=αL bin +βL loc

[0049] In the formula, α and β are the weight coefficients of the loss function, which are used to balance the text binarization loss and positioning loss.

[0050] The text binarization loss is:

[0051]

[0052] In the formula, is the binary text image generated by the network, yi is the labeled binary text image, and N is the number of samples.

[0053] The localization loss (deformable convolution loss) is:

[0054]

[0055] In the formula, is the text position generated by the network, b i is the marked text position, and N is the number of samples.

[0056] During the training process, the back propagation algorithm is used to optimize the overall loss function. In the forward propagation process, the output of the network is calculated by inputting data and the current network parameters. In the back propagation process, the gradient of the loss function to the output is first calculated, and then the gradient of the loss function to the network parameters is calculated layer by layer using the chain rule.

[0057] S22, input the text area into the text recognition network CRNN to recognize the text content. The text recognition network CRNN includes three parts: feature extraction, sequence modeling and transcription. In CRNN, CNN is used for feature extraction and RNN is used for sequence modeling.

[0058] CNN extracts local features from the input image data through convolution operations, and convolves the image through a series of convolution kernels to capture local feature information such as image texture and edges:

[0059]

[0060] Where F is the feature map, σ is the activation function, (x, y) is the coordinate of the feature map, m×n is the size of the convolution kernel, I(x+i, y+j) is the pixel value of the input image at position (x+i, y+j), K(i, j) is the weight of the convolution kernel at position (i, j), and b is the bias term.

[0061] RNN updates the hidden state based on the input features at the current moment and the hidden state at the previous moment. This recursive update process enables RNN to capture long-term dependencies in the sequence and encode this information into the hidden state, thereby modeling the sequence information. The hidden state and output of RNN are expressed as:

[0062] H t =tanh(W ht X t +W hh H t-1 +b h )

[0063] Y t =softmax(W yh H t +b y )

[0064] Among them, Ht is the hidden layer state of RNN at time t, Xt is the input feature of the input sequence at time t, Whx is the weight matrix from the input layer to the hidden layer, Whh is the weight matrix from the hidden layer to other hidden layers, bh is the bias vector of the hidden layer, and tanh is the hyperbolic tangent activation function. Yt represents the output of RNN at time t, Wyh is the weight matrix from the hidden layer to the output layer, by is the bias vector of the output layer, and softmax is the activation function for multi-classification tasks.

[0065] CRNN combines CNN and RNN to achieve joint modeling of image and sequence data. Its end-to-end structure enables the model to learn and infer directly from raw data without manual feature design.

[0066] S23. Use the Chinese Word Segmentation Engine (CSWE) to split the recognized text into short words, and compare the short words with the target phrase to determine whether the recognized text content contains the target phrase. If the short words after division exist in the Word2Vec corpus, they are mapped into space vectors and the semantic distance similarity is calculated with the target phrase using cosine similarity. The closer the cosine similarity value is to 1, the more similar the two words are semantically. If the short words after division do not exist in the Word2Vec corpus, the edit distance similarity is calculated with the target phrase.

[0067] Cosine similarity measures the cosine value of the angle between two vectors to determine how similar their directions are in vector space:

[0068]

[0069] The edit distance algorithm is a method to measure the similarity between two strings, that is, to calculate the minimum number of single-character edits required to convert from one string to another, including operations such as inserting, deleting, and replacing characters. The similarity of sentences is inversely proportional to the number of edit operations:

[0070] Sim(s, t) = 1 - LD / Lmax(s, t)

[0071] Where LD represents the edit distance and Lmax(s,t) represents the maximum length of two sentences.

[0072] If there are two strings S(s1,s2,…,sm) and T(t1,t2,…,tn), let the matrix LD[m+1,n+1] be used. The algorithm loops through the values ​​of each cell LD(i,j) in the matrix. Then the edit distance LD between the strings S and T is calculated by equations (19) and (20)

[22] :

[0073]

[0074] Min=min{LD(i-1,j)+1, LD(i,j-1)+1, LD(i-1,j-1)+f(i,j)

[0075] Where, when the i-th word of S is not equal to the j-th word of T, f(i,j)=1; otherwise, f(i,j)=0.

[0076] The multimodal semantic aggregation module performs weighted summation of image confidence and text similarity, adjusts the weights according to the confidence and similarity of each modality, and reduces the impact of incomplete or unclear information in text or images by comprehensively utilizing multimodal information.

[0077] The multimodal semantic aggregation module outputs the detection results according to the recognition confidence of image recognition and text recognition. The specific working process of the multimodal semantic aggregation module is as follows:

[0078] S31. If the recognition confidence of one modality among the image and text is higher than the preset high threshold, and the confidence of the other modality is lower than the preset low threshold, directly output the recognition result of the single modality with higher confidence. If this situation does not occur, enter S32.

[0079] S32, performing weighted summation of the recognition confidences of the image recognition and the text recognition to calculate the multimodal confidence. During the calculation process, the weights are adjusted according to the recognition confidence of each modality, and the impact of incomplete or unclear text or image information is reduced by comprehensively utilizing multimodal information.

[0080] Assume that the recognition confidence of image recognition is CI, the recognition confidence of text recognition is ST, the weight of image modality is WI, and the weight of text modality is WT. The weighted summation formula is as follows:

[0081] F=w I ·C I +w T ·S T

[0082] Among them, WI and WT satisfy WI+WT=1.

[0083] In order to optimize the weight parameters, the present invention introduces a five-fold cross-validation method, which divides the data set D into five mutually exclusive subsets D1, D2, D3, D4, and D5, selects one of them as the validation set, and the remaining four as the training set. The process is repeated five times to ensure that each subset is validated. The core of the method is to compare the performance indicators of each cross-validation under different weight parameters to select the best weight parameters.

[0084] S33. If the multimodal confidence reaches the preset accuracy, the target location information is output and it is marked as containing a graphic target. If the multimodal confidence does not reach the preset accuracy, it will be reported that the target is not detected.

[0085] We took photos of no-smoking signs in public places in the city, built a corresponding data set, and conducted comparative experiments. The self-created data set covers the changes of the target under various scenes and angles. Before image annotation, the collected images were uniformly scaled to a resolution of 640×640 pixels to ensure that more details and information in the image were retained. The data set has a total of 9290 images, which are divided into training set, test set, and validation set in a ratio of 7:2:1. There are 6600 images in the training set, 2000 images in the test set, and 690 images in the validation set. The experiment was conducted using the PyTorch deep learning framework, and the hardware used in the experiment was a GeForce RTX 3060 with 12GB video memory. Public data sets and Baidu image search were used for text recognition training, including LSVT, RCTW-17, MTWI 2018, CASIA-10K, and SROIE, and the ratio of training to validation set was set to 8.5:1.5. The Chinese word segmentation engine CSWE is introduced to preprocess the text recognition results to solve the problem of semantic analysis difficulties of long Chinese sentences through word segmentation and semantic unit extraction. At the same time, the RandAugment data enhancement operation is used to improve the directional classification effect of the image. When calculating the semantic distance similarity, Word2Vec is used for text vectorization. However, some uncommon expressions of no-smoking signs are not included in the corpus, so these missing expressions are split and their average vector values ​​are calculated, and then these new vectors are introduced into the corpus. For example, "Thanks for not smoking" is not included in the corpus. By calculating the average vector of "thank you", "no" and "smoking", the spatial vector of this expression is generated and added to the corpus. The comparative experimental results show that in both image and text fields, the dual-modal image-text fusion target recognition system has achieved high accuracy, among which the image recognition accuracy is 95.58% and the text recognition accuracy is 92%. At the same time, the recognition accuracy under dual mode reaches 97.8%, which is 2.22% and 3.73% higher than that of single modality recognition technology, respectively.

[0086] The dual-modal image-text fusion recognition method based on the dual-modal image-text fusion target recognition system specifically includes the following steps:

[0087] S1, the image recognition module uses an improved YOLOv5-based image recognition model for target detection;

[0088] S2, the text recognition module determines whether the recognized text content contains the target phrase;

[0089] S3, the multimodal semantic aggregation module outputs the detection results according to the recognition confidence of image recognition and text recognition.

[0090] The above disclosure is only the preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the present invention. It should be pointed out that for those skilled in the art, any equivalent changes made to the scheme of the present invention without departing from the design structure and principle of the present invention are considered to be within the protection scope of the present invention.

Claims

1. A dual-modal image-text fusion target recognition system, characterized in that ,The dual-modal image-text fusion target recognition system includes an image recognition module, a text recognition module and a multi-modal semantic aggregation module, and the multi-modal semantic aggregation module is respectively connected to the image recognition module and the text recognition module; The image recognition module uses an image recognition model to perform target detection; The text recognition module is used to determine whether the recognized text content contains a target phrase; The multimodal semantic aggregation module outputs detection results according to the recognition confidence of image recognition and text recognition.

2. The dual-modal image-text fusion target recognition system according to claim 1, characterized in that: The image recognition model is improved based on YOLOv5, including an input end Input, a feature extraction network Backbone, a feature fusion network Neck and a prediction end Head. The feature fusion network Neck includes a small target detection layer of 160×160 pixels, and the C3 module of the small target detection layer includes a convolution block attention module.

3. The dual-modal image-text fusion target recognition system according to claim 2, characterized in that: The prediction end Head adopts a decoupled detection head.

4. The dual-modal image-text fusion target recognition system according to claim 3, characterized in that: The recognition process of the image recognition model is as follows: S11, the features of the 80×80 scale feature layer in the feature extraction network Backbone and the up-sampled feature layer in the feature fusion network Neck are processed by the C3 module to obtain a deep semantic feature layer containing the feature information of the small target; S12, concatenate and fuse the deep semantic feature layer with the shallow position feature layer in the feature extraction network Backbone to improve the 160×160 scale fusion feature layer; S13, 160×160 scale fusion feature layer is processed by C3 module and sent to the prediction end Head; S14. Regenerate 12 preset anchor box scales based on the K-means clustering algorithm, including 3 scales each for large, medium, small and extremely small targets.

5. The dual-modal image-text fusion target recognition system according to claim 1, characterized in that: The specific working process of the text recognition module is as follows: S21, input the image to be detected into the text detection model DB to determine the text area; S22, inputting the text area into the text recognition network CRNN to recognize the text content; S23. Use a Chinese word segmentation engine to split the recognized text into short words, and compare the short words with the target phrase to determine whether the recognized text content contains the target phrase.

6. The dual-modal image-text fusion target recognition system according to claim 5, characterized in that: During the training process of the text detection model DB, text binarization loss and positioning loss are used as the main optimization targets, and the back propagation algorithm is used to optimize the overall loss function.

7. The dual-modal image-text fusion target recognition system according to claim 5, characterized in that: In S23, if the divided short word exists in the Word2Vec corpus, it is mapped into a space vector, and the semantic distance similarity is calculated with the target phrase using cosine similarity. The closer the cosine similarity value is to 1, the more similar the two words are semantically. If the divided short word does not exist in the Word2Vec corpus, the edit distance similarity is calculated between it and the target phrase.

8. The dual-modal image-text fusion target recognition system according to claim 1, characterized in that: The specific working process of the multimodal semantic aggregation module is as follows: S31, if the recognition confidence of one modality among the image and the text is higher than the preset high threshold, and the confidence of the other modality is lower than the preset low threshold, directly output the recognition result of the single modality with higher confidence; if this situation does not occur, enter S32; S32, performing weighted summation on the recognition confidence of the image recognition and the text recognition to calculate a multimodal confidence; S33. If the multimodal confidence reaches the preset accuracy, the target location information is output and it is marked as containing a graphic target. If the multimodal confidence does not reach the preset accuracy, it will be reported that the target is not detected.

9. The dual-modal image-text fusion target recognition system according to claim 8, characterized in that: In S32, a five-fold cross validation method is used to optimize the weight parameters.

10. A dual-modal image-text fusion recognition method based on the dual-modal image-text fusion target recognition system according to any one of claims 1 to 9, characterized in that ,The dual-modal image-text fusion recognition method specifically includes the following steps: S1, the image recognition module uses an improved YOLOv5-based image recognition model for target detection; S2, the text recognition module determines whether the recognized text content contains the target phrase; S3, the multimodal semantic aggregation module outputs the detection results according to the recognition confidence of image recognition and text recognition.

Images