Intelligent aviation mark identification method and system based on multiple visual modes

Abstract

The invention provides an intelligent aviation mark identification method and system based on multiple visual modes. The method comprises the following steps: acquiring a to-be-identified aircraft image; according to the to-be-recognized aircraft image, cooperatively utilizing a target detection model and a text recognition model to carry out target detection and text recognition to obtain an identifier detection result and an identifier recognition result, the text recognition model being used for carrying out feature extraction on the input to-be-recognized aircraft image; performing text recognition on the extracted text features based on an identification detection result obtained by the target detection model to obtain an identification recognition result; the target detection model is used for carrying out feature extraction on an input to-be-recognized aircraft image and carrying out multi-modal feature fusion based on the extracted image features and text features so as to carry out target detection on the fused features and obtain an identification detection result. According to the method, the target detection model and the text recognition model are cooperatively utilized, so that multi-task efficient cooperative processing is realized, the robustness of the models in different scenes is improved, and misinformation and missing report conditions are reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an intelligent aviation sign recognition method and system based on visual multimodality. Background Art

[0002] In recent years, computer vision and deep learning technologies have made significant progress in the field of image and video processing, promoting the development of many practical applications such as security monitoring, autonomous driving, and drone navigation. These technologies have achieved efficient detection and recognition of objects in images and videos through deep neural networks (DNNs) and convolutional neural networks (CNNs). However, existing technologies still face many challenges in complex environments, especially in real-time video stream processing and high-precision recognition tasks.

[0003] At present, target detection technology has evolved from traditional feature-based methods (such as SIFT, HOG) to deep learning-based methods (such as R-CNN, YOLO, SSD). These methods use deep convolutional neural networks to train on large-scale data sets and can accurately detect and locate targets in complex backgrounds. Optical character recognition (OCR) technology is used to extract text information from images. Traditional methods rely on template matching and feature extraction, while modern methods mainly rely on deep learning models such as LSTM and Transformer. OCR is widely used in various application scenarios, such as document digitization and license plate recognition.

[0004] However, the above technology performs poorly in complex scenarios, such as those with different lighting conditions and cluttered backgrounds, and suffers from problems such as false alarms and missed alarms. In addition, its multi-task processing capabilities are insufficient, making it difficult to complete target detection and character recognition at the same time. In addition, real-time detection is often limited by hardware performance and cannot guarantee efficient real-time processing. The model also requires large amounts of calculation, which affects real-time performance. Summary of the invention

[0005] The present invention provides an intelligent aviation sign recognition method and system based on visual multimodality, which is used to solve the defect in the prior art that the performance under complex fields is not improved, which affects the recognition accuracy, improve the robustness of the model in different scenarios, and reduce false positives and negative negatives.

[0006] The present invention provides an intelligent aviation identification method based on visual multimodality, comprising: obtaining an aircraft image to be identified; according to the aircraft image to be identified, collaboratively using a target detection model and a text recognition model to perform target detection and text recognition to obtain a identification detection result and a identification recognition result; wherein the target detection model and the text recognition model are obtained by training based on aircraft image training data and identification detection labels and identification recognition labels corresponding to the aircraft image training data; the text recognition model is used to perform feature extraction on the input aircraft image to be identified, and perform text recognition on the extracted text features based on the identification detection result obtained by the target detection model to obtain the identification recognition result; the target detection model is used to perform feature extraction on the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain the identification detection result.

[0007] According to an intelligent aviation sign recognition method based on visual multimodality provided by the present invention, the text recognition model includes a text feature extraction layer and a text recognition layer, and the target detection model includes an image feature extraction layer, a feature fusion layer and a target detection layer, wherein: the text feature extraction layer, based on the attention mechanism, performs feature extraction on the input aircraft image to be recognized to obtain text features; the text recognition layer, based on the sign detection result obtained by the target detection model, performs text recognition on the text features to obtain the sign recognition result; the image feature extraction layer, performs feature extraction on the input aircraft image to be recognized to obtain image features; the feature fusion layer, based on the feature pyramid, performs multimodal feature fusion on the text features and the image features to obtain the pyramid fusion features; the target detection layer, performs target detection on the fused features to obtain the sign detection result.

[0008] According to a method for intelligent aviation sign recognition based on visual multimodality provided by the present invention, based on the attention mechanism, feature extraction is performed on an input aircraft image to be recognized to obtain text features, including: feature extraction is performed on the input aircraft image to be recognized to obtain a feature vector; position encoding is performed on the feature vector; based on the multi-head attention mechanism and position encoding, the self-attention vector between the feature vector of each position and the feature vectors of other positions is determined to obtain a context representation vector corresponding to the feature vector of each position; using a feedforward neural network, the context representation vector of the feature vector corresponding to each position is mapped into a vector space to obtain a coded feature; the coded feature is decoded to obtain a text feature.

[0009] According to an intelligent aviation sign recognition method based on visual multimodality provided by the present invention, text recognition is performed on text features based on the sign detection result obtained by the target detection model to obtain the sign recognition result, including: based on the sign detection result currently obtained by the target detection model, selecting the text feature in the corresponding detection box and the sign recognition result obtained once, performing probability distribution prediction on the next character to obtain the corresponding predicted probability distribution, and selecting the character with the highest predicted probability distribution as the corresponding sign recognition result; cyclically performing probability distribution prediction on the next character and selecting the character with the highest predicted probability distribution as the corresponding sign recognition result until the sum of the character lengths of all the sign recognition results obtained meets the preset length; or, based on the sign detection result currently obtained by the target detection model, selecting the text feature in the corresponding detection box, and classifying the selected text features to map each text feature to the corresponding character category to obtain the sign recognition result.

[0010] According to a method for intelligent aviation identification based on visual multimodality provided by the present invention, feature extraction is performed on an input aircraft image to be identified to obtain image features, including: extracting features from the input aircraft image to be identified layer by layer from low layers to high layers to obtain image features; based on a feature pyramid, multimodal feature fusion is performed on text features and image features to obtain pyramid fusion features, including: convolving and pooling the image features extracted from the first layer from high layers to low layers to obtain a first feature; upsampling the image features extracted from the first layer, and performing feature fusion of the obtained upsampled features with the image features extracted from the second layer to obtain corresponding fusion features; wherein the level of the first layer is higher than that of the second layer; upsampling the fusion features, and performing feature fusion of the obtained upsampled features with the image features extracted from the corresponding lower layer to obtain corresponding fusion features; repeatedly upsampling the fusion features, and performing feature fusion of the obtained upsampled features with the image features extracted from the corresponding lower layer until the image features extracted from the lowest layer are fused with the upsampled features from the corresponding higher layer, and obtaining pyramid fusion features according to the first feature and all the obtained corresponding fusion features.

[0011] According to a method for intelligent aviation sign recognition based on visual multimodality provided by the present invention, after performing target detection and text recognition according to an aircraft image to be recognized by using a target detection model and a text recognition model to obtain a sign detection result and a sign recognition result, the method comprises: determining a confidence threshold according to the confidence of the sign detection result and the context information of the sign recognition result, and screening the sign detection result by using the confidence threshold; and / or removing redundant detection frames in the sign detection result by using non-maximum suppression; and / or correcting the sign recognition result based on preset character legitimacy rules and preset format rules; and / or inputting the sign detection results and the sign recognition results corresponding to each aircraft image to be recognized within a target time into a result correction model to obtain a correction result output by the result correction model; wherein the result correction model is obtained by training based on aircraft image training data within a historical time period and the corresponding sign detection true value and sign recognition true value.

[0012] According to a method for intelligent aviation sign recognition based on visual multimodality provided by the present invention, before performing target detection and text recognition according to an aircraft image to be recognized by using a target detection model and a text recognition model to obtain a sign detection result and a sign recognition result, the method comprises: obtaining aircraft image training data and sign detection labels and sign recognition labels corresponding to the aircraft image training data; using the aircraft image training data as input data for training the target detection model to be trained, using the sign detection labels corresponding to the aircraft image training data as labels for training the target detection model to be trained, and using the aircraft image training data as input data for training the text recognition model to be trained, using the sign recognition labels corresponding to the aircraft image training data as labels for training the text recognition model to be trained, and using a collaborative mechanism to train the target detection model to be trained and the text recognition model to be trained in parallel.

[0013] The present invention also provides an intelligent aviation identification system based on visual multimodality, comprising: a data acquisition module, which acquires an aircraft image to be identified; a identification identification module, which performs target detection and text recognition based on the aircraft image to be identified by collaboratively utilizing a target detection model and a text recognition model to obtain a identification detection result and a identification recognition result; wherein the target detection model and the text recognition model are obtained by training based on aircraft image training data and identification detection labels and identification recognition labels corresponding to the aircraft image training data; the text recognition model is used to perform feature extraction on the input aircraft image to be identified, and perform text recognition based on the extracted text features to obtain a identification recognition result; the target detection model is used to perform feature extraction on the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain a identification detection result.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the intelligent aviation sign recognition method based on visual multimodality as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the intelligent aviation sign recognition method based on visual multimodality as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned intelligent aviation sign recognition methods based on visual multimodality.

[0017] The intelligent aviation sign recognition method and system based on visual multimodality provided by the present invention realizes efficient collaborative processing of multiple tasks by collaboratively utilizing a target detection model and a text recognition model, and effectively combines the image features extracted by the target detection model and the text features extracted by the text recognition model to accurately detect the target representation, and uses the sign detection results obtained by the detection to recognize the text features, thereby using the sign detection results to provide contextual information for text recognition, helping text recognition to better understand the position and meaning of the text in the image, further improving the robustness of the model in different scenarios, and improving the overall recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the intelligent aviation sign recognition method based on visual multimodality provided by the present invention; Figure 2 It is a schematic diagram of the architecture of the intelligent aviation sign recognition method based on visual multimodality provided by the present invention; Figure 3 It is a structural schematic diagram of an intelligent aviation sign recognition system based on visual multimodality provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Figure 1 is a flow chart of the intelligent aviation identification method based on visual multimodality provided by the present invention, such as Figure 1 As shown, the method includes the following: S11, obtaining an image of an aircraft to be identified; S12, according to the aircraft image to be identified, collaboratively use the target detection model and the text recognition model to perform target detection and text recognition to obtain the identification detection result and the identification recognition result; wherein, the target detection model and the text recognition model are trained based on the aircraft image training data and the identification detection labels and identification recognition labels corresponding to the aircraft image training data; the text recognition model is used to extract features of the input aircraft image to be identified, and perform text recognition on the extracted text features based on the identification detection result obtained by the target detection model to obtain the identification recognition result; the target detection model is used to extract features of the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain the identification detection result.

[0022] It should be noted that the step numbers "S1N" in this manual do not represent the order of the intelligent aviation sign recognition method based on visual multimodality. Figure 2 The present invention describes an intelligent aviation sign recognition method based on visual multimodality.

[0023] Step S11, obtaining an image of an aircraft to be identified.

[0024] In an optional embodiment, after acquiring the image of the aircraft to be identified, the method includes: using a preset filtering technology to perform denoising on the image of the aircraft to be identified to remove high-frequency noise in the image; and using an adaptive sharpening technology to sharpen the denoised image of the aircraft to be identified to enhance image details and improve image clarity. It should be added that the adaptive filtering technology can be selected according to actual design requirements, such as Gaussian filtering and median filtering, which are not further limited here.

[0025] In addition, after obtaining the image of the aircraft to be identified, the process also includes: performing white balance adjustment and color correction on the image to be identified, thereby ensuring the consistency of image color and reducing the impact of lighting changes.

[0026] Step S12, according to the aircraft image to be identified, collaboratively use the target detection model and the text recognition model to perform target detection and text recognition to obtain the identification detection result and the identification recognition result; wherein, the target detection model and the text recognition model are obtained by training based on the aircraft image training data and the identification detection labels and identification recognition labels corresponding to the aircraft image training data; the text recognition model is used to extract features of the input aircraft image to be identified, and perform text recognition on the extracted text features based on the identification detection results obtained by the target detection model to obtain the identification recognition result; the target detection model is used to extract features of the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain the identification detection result.

[0027] Accordingly, according to the image of the aircraft to be identified, the target detection model and the text recognition model are collaboratively used to perform target detection and text recognition to obtain a logo detection result and a logo recognition result, including: inputting the image of the aircraft to be identified into the target detection model and the text recognition model respectively, so as to collaboratively use the target detection model and the text recognition model to obtain a logo detection result and a logo recognition result.

[0028] In an optional embodiment, before inputting the aircraft image to be identified into the text recognition model, it also includes: converting the aircraft image to be identified into a grayscale image; converting the grayscale image into a black and white binary image through a preset threshold to obtain foreground text and background; and denoising the black and white binary image.

[0029] It should be noted that by converting the color image of the aircraft to be identified into a grayscale image, the main features of the image are retained while reducing the amount of calculation, and the pixels in the image are divided into foreground text and background by a preset threshold to highlight the text area, thereby improving the subsequent text recognition model's ability to extract aviation identification text information in the aircraft image, so that the model can focus more on the analysis and recognition of text features, and improve the accuracy and efficiency of text recognition. In addition, denoising can further remove interference factors in the image, make the text area clearer, and reduce the impact of noise on feature extraction and text recognition.

[0030] In an optional embodiment, the target detection model uses convolutional neural networks (CNN) to simultaneously process targets of different scales, and introduces a feature pyramid network (FPN) to enhance the model's ability to detect small targets. In addition, the text recognition model uses optical character recognition (OCR) with an attention mechanism (Transformer), which is used to implement sequence-to-sequence (Seq2Seq) model tasks, improve the recognition accuracy of long texts and irregular texts, and enhance the model's character recognition capabilities in complex backgrounds.

[0031] In an optional embodiment, the coordinated use of the target detection model and the text recognition model for target detection and text recognition can be achieved through a pre-built parallel multi-model architecture, so as to accelerate the reasoning process of the model through parallel computing technology and ensure the real-time performance of the system. It should be noted that when building a parallel multi-model architecture, it is necessary to fully consider the communication between the target detection model and the text recognition model, and dynamically allocate computing resources to ensure that the system can still run efficiently under high load conditions, thereby realizing a load balancing strategy.

[0032] In this embodiment, reference Figure 2 The text recognition model includes a text feature extraction layer and a text recognition layer, and the target detection model includes an image feature extraction layer, a feature fusion layer and a target detection layer, wherein: the text feature extraction layer, based on the attention mechanism, performs feature extraction on the input aircraft image to be identified to obtain text features; the text recognition layer, based on the logo detection result obtained by the target detection model, performs text recognition on the text features to obtain logo recognition results; the image feature extraction layer, performs feature extraction on the input aircraft image to be identified to obtain image features; the feature fusion layer, based on the feature pyramid, performs multimodal feature fusion on the text features and image features to obtain pyramid fusion features; the target detection layer, performs target detection on the fused features to obtain logo detection results.

[0033] It should be noted that by extracting text features based on the attention mechanism, focusing on important areas related to the text in the image, more representative text features can be effectively extracted to reduce the interference of irrelevant information; by extracting image features, various details and semantic information of the image can be fully utilized to provide a rich feature basis for target detection; by using the feature pyramid to fuse text features and image features for multimodal feature fusion, the fused features contain both global and local information of the image and specific information of the text, so as to better cope with complex situations in the image, such as illumination changes, noise interference, partial occlusion, etc., which helps to detect targets more accurately, and use the obtained logo detection results to identify text features, so as to use the logo detection results to provide contextual information for text recognition, help the text recognition layer to better understand the position and meaning of the text in the image, further improve the accuracy and robustness of text recognition, and improve the overall recognition accuracy.

[0034] Specifically, based on the attention mechanism, feature extraction is performed on the input aircraft image to be identified to obtain text features, including: feature extraction is performed on the input aircraft image to be identified to obtain feature vectors; position encoding is performed on the feature vectors; based on the multi-head attention mechanism and position encoding, the self-attention vector between the feature vector of each position and the feature vectors of other positions is determined to obtain the context representation vector of the feature vector corresponding to each position; using a feedforward neural network, the context representation vector of the feature vector corresponding to each position is mapped to the vector space to obtain the encoding feature; the encoding feature is decoded to obtain the text feature. It should be added that a residual connection is set between the output and input of each multi-head attention and feedforward neural network and layer normalization is performed to improve the stability of the model.

[0035] Furthermore, based on the multi-head attention mechanism and position encoding, the self-attention vector between the feature vector of each position and the feature vectors of other positions is determined, including: based on the multi-head attention mechanism, for each attention head, a linear transformation is performed on the feature vector after position encoding to obtain the corresponding query vector, key vector and value vector; according to the query vector and the key vector, an attention weight is obtained; based on the attention weight and the value vector, an output vector of the corresponding attention head is obtained; the output vectors of multiple attention heads are concatenated and linearly transformed to obtain the self-attention vector between the feature vector of each position of the multi-head self-attention output and the feature vectors of other positions.

[0036] In addition, based on the identification detection result obtained by the target detection model, text recognition is performed on the text features to obtain the identification recognition result, including: based on the identification detection result currently obtained by the target detection model, the text features in the corresponding detection box and the identification recognition result obtained previously are selected, and the probability distribution of the next character is predicted to obtain the corresponding predicted probability distribution, and the character with the highest predicted probability distribution is selected as the corresponding identification recognition result; the probability distribution prediction of the next character is cyclically performed and the character with the highest predicted probability distribution is selected as the corresponding identification recognition result until the sum of the character lengths of all the identification recognition results obtained meets the preset length; or, based on the identification detection result currently obtained by the target detection model, the text features in the corresponding detection box are selected, and the selected text features are classified to map each text feature to the corresponding character category to obtain the identification recognition result.

[0037] It should be noted that by selecting the text features in the corresponding detection box according to the identification detection results to predict the probability distribution of the next character, it is possible to more accurately focus on the text area related to the target identification, reduce the interference of irrelevant text, and thus improve the accuracy of text recognition. By cyclically predicting the probability distribution and selecting the highest probability character as the result, the identification recognition result is gradually constructed, making the recognition process more refined and accurate, and being able to better handle complex text sequences. In addition, by selecting the text features in the corresponding detection box according to the identification detection results for classification, the text features are mapped to character categories using the context information provided by the target detection, and the accuracy of classification is improved, thereby improving the overall quality of the text recognition results.

[0038] In an optional embodiment, after obtaining the identification recognition result, it also includes: based on preset character legality rules and preset format rules, correcting the identification recognition result to remove repeated characters, correct character or format errors, etc., to avoid repeated recognition of the same character, improve the standardization and usability of the result, and make the final result more accurate.

[0039] In an optional embodiment, extracting features from the input aircraft image to be identified to obtain image features includes: extracting features layer by layer from the input aircraft image to be identified from a low level to a high level to obtain image features.

[0040] Correspondingly, based on the feature pyramid, multimodal feature fusion of text features and image features is performed to obtain pyramid fusion features, including: from high level to low level, convolving and pooling the image features extracted from the first layer to obtain the first feature; upsampling the image features extracted from the first layer, and performing feature fusion of the obtained upsampled features with the image features extracted from the second layer to obtain the corresponding fusion features; wherein the level of the first layer is higher than the second layer; upsampling the fusion features, and performing feature fusion of the obtained upsampled features with the image features extracted from the corresponding lower layer to obtain the corresponding fusion features; repeatedly upsampling the fusion features, and performing feature fusion of the obtained upsampled features with the image features extracted from the corresponding lower layer, until the image features extracted from the lowest layer are completely fused with the upsampled features from the corresponding higher layer, and the pyramid fusion features are obtained according to the first feature and all the corresponding fusion features obtained.

[0041] It should be noted that the first feature is obtained by convolving and pooling high-level image features to extract more abstract and more global semantic information, which helps to grasp the overall structure and main targets of the image, and combines upsampling and feature fusion from high to low layers to make features at different levels complement each other, making full use of the global context of high-level features and the detailed information of low-level features to obtain pyramid fusion features with multi-scale characteristics, thereby better adapting to targets of different sizes on the basis of improving the ability to understand and analyze images.

[0042] In an optional embodiment, after performing target detection and text recognition based on the aircraft image to be identified using the target detection model and the text recognition model to obtain the logo detection result and the logo recognition result, it includes: determining a confidence threshold based on the confidence of the logo detection result and the context information of the logo recognition result, and using the confidence threshold to filter the logo detection result.

[0043] It should be added that the confidence threshold can be determined based on the confidence of the identification detection result and the context information of the identification recognition result, combined with prior experience or threshold preset rules. The threshold preset rules can be set according to prior experience or actual design requirements to limit the thresholds corresponding to the required configurations of different confidence levels and context information, which are not further defined here. In addition, by using the confidence threshold, more reliable detection results can be screened out, thereby eliminating inaccurate or false detection results and improving overall accuracy.

[0044] In an optional embodiment, after performing target detection and text recognition based on the aircraft image to be identified using a target detection model and a text recognition model to obtain a logo detection result and a logo recognition result, it also includes: using non-maximum suppression to remove redundant detection boxes in the logo detection result to improve the accuracy of the detection result.

[0045] In an optional embodiment, after performing target detection and text recognition based on the aircraft image to be identified using the target detection model and the text recognition model to obtain the identification detection result and the identification recognition result, it also includes: inputting the identification detection result and the identification recognition result corresponding to each aircraft image to be identified within the target time into the result correction model to obtain the correction result output by the result correction model; wherein the result correction model is trained based on the aircraft image training data within the historical time period and the corresponding identification detection true value and identification recognition true value.

[0046] It should be noted that the result correction model can select the corresponding machine learning algorithm according to the actual design requirements, such as Bayesian optimization, to dynamically adjust the logo recognition results and improve the accuracy of text recognition.

[0047] In an optional embodiment, before performing target detection and text recognition using a target detection model and a text recognition model based on an aircraft image to be recognized and obtaining a mark detection result and a mark recognition result, the method includes: obtaining aircraft image training data and a mark detection label and a mark recognition label corresponding to the aircraft image training data; using the aircraft image training data as input data for training the target detection model to be trained, using the mark detection labels corresponding to the aircraft image training data as labels for training the target detection model to be trained, and using the aircraft image training data as input data for training the text recognition model to be trained, using the mark recognition labels corresponding to the aircraft image training data as labels for training the text recognition model to be trained, and using a collaborative mechanism to train the target detection model to be trained and the text recognition model to be trained in parallel.

[0048] Specifically, obtaining aircraft image training data and logo detection labels and logo recognition labels corresponding to the aircraft image training data includes: based on diversified data sources, collecting aircraft images under different lighting (daytime, nighttime) and weather conditions to obtain aircraft image training data; annotating the collected aircraft images to obtain logo detection labels and logo recognition labels.

[0049] It should be added that the diverse data sources include at least one of fixed cameras, drones and satellite images, the lighting includes lighting in different time periods, such as day and night, and the weather conditions include sunny, cloudy, rainy, foggy and snowy days. In addition, the annotations include information such as the location, type, attitude, size, etc. of the aircraft to ensure the high quality and diversity of the data set.

[0050] In an optional embodiment, after obtaining the aircraft image training data, the method further includes: performing image enhancement on the aircraft image training data, such as histogram equalization, CLAHE (contrast limited adaptive histogram equalization), etc., and automatically adjusting according to the illumination and contrast of the image.

[0051] In an optional embodiment, after acquiring the image of the aircraft to be identified, the method includes: using a preset filtering technology to perform denoising on the image of the aircraft to be identified to remove high-frequency noise in the image; and using an adaptive sharpening technology to sharpen the denoised image of the aircraft to be identified to enhance image details and improve image clarity. It should be added that the adaptive filtering technology can be selected according to actual design requirements, such as Gaussian filtering and median filtering, which are not further limited here.

[0052] In addition, after obtaining the image of the aircraft to be identified, the process also includes: performing white balance adjustment and color correction on the image to be identified, thereby ensuring the consistency of image color and reducing the impact of lighting changes.

[0053] Furthermore, after the aircraft image training data, it also includes: based on a preset data enhancement strategy, data enhancement is performed on the aircraft image training data; wherein the preset data enhancement strategy includes image distortion, text occlusion and noise addition, etc., and / or the preset data enhancement strategy also includes random cropping, scaling, etc., to improve the robustness of the model.

[0054] In an optional embodiment, in the process of parallel training of the target detection model to be trained and the text recognition model to be trained, model pruning and quantization techniques can also be used to reduce the number of model parameters, improve the inference speed, and adapt to the deployment requirements of resource-constrained devices.

[0055] In summary, the embodiments of the present invention achieve efficient collaborative processing of multiple tasks by collaboratively utilizing the target detection model and the text recognition model, and effectively combine the image features extracted by the target detection model and the text features extracted by the text recognition model to accurately detect the target representation, and use the identification detection results obtained by the detection to identify the text features, thereby using the identification detection results to provide contextual information for text recognition, helping text recognition to better understand the position and meaning of the text in the image, further improving the robustness of the model in different scenarios, and improving the overall recognition accuracy.

[0056] The intelligent aviation sign recognition system based on visual multimodality provided by the present invention is described below. The intelligent aviation sign recognition system based on visual multimodality described below and the intelligent aviation sign recognition method based on visual multimodality described above can be referenced to each other.

[0057] Figure 3 The schematic diagram of the structure of an intelligent aviation identification system based on visual multimodality is shown, the system comprising: A data acquisition module 31 acquires an image of an aircraft to be identified; The identification recognition module 32 performs target detection and text recognition based on the aircraft image to be recognized by collaboratively utilizing the target detection model and the text recognition model to obtain the identification detection result and the identification recognition result; wherein the target detection model and the text recognition model are obtained by training based on the aircraft image training data and the identification detection labels and identification recognition labels corresponding to the aircraft image training data; the text recognition model is used to perform feature extraction on the input aircraft image to be recognized, and perform text recognition on the extracted text features based on the identification detection result obtained by the target detection model to obtain the identification recognition result; the target detection model is used to perform feature extraction on the input aircraft image to be recognized, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain the identification detection result.

[0058] In an optional embodiment, the system further includes: a filtering module, after acquiring the image of the aircraft to be identified, using a preset filtering technology to perform denoising on the image of the aircraft to be identified, so as to remove high-frequency noise in the image; a sharpening module, using an adaptive sharpening technology to sharpen the denoised image of the aircraft to be identified, so as to enhance image details and improve image clarity. It should be added that the adaptive filtering technology can be selected according to actual design requirements, such as Gaussian filtering and median filtering, which are not further limited here.

[0059] In addition, the system also includes: a color correction module, after obtaining the image of the aircraft to be identified, performs white balance adjustment and color correction on the image to be identified, so as to ensure the consistency of image color and reduce the impact of lighting changes.

[0060] In this embodiment, the logo recognition module 32 is used to input the aircraft image to be recognized into the target detection model and the text recognition model respectively, so as to collaboratively utilize the target detection model and the text recognition model to obtain the logo detection result and the logo recognition result.

[0061] In an optional embodiment, the system further includes: a grayscale processing module, which converts the aircraft image to be identified into a grayscale image before inputting the aircraft image to be identified into a text recognition model; a binary image processing module, which converts the grayscale image into a black and white binary image through a preset threshold to obtain foreground text and background; and a denoising module, which performs denoising on the black and white binary image.

[0062] In an optional embodiment, the coordinated use of the target detection model and the text recognition model for target detection and text recognition can be achieved through a pre-built parallel multi-model architecture, so as to accelerate the reasoning process of the model through parallel computing technology and ensure the real-time performance of the system. It should be noted that when building a parallel multi-model architecture, it is necessary to fully consider the communication between the target detection model and the text recognition model, and dynamically allocate computing resources to ensure that the system can still run efficiently under high load conditions, thereby realizing a load balancing strategy.

[0063] In this embodiment, the text recognition model includes a text feature extraction layer and a text recognition layer, and the target detection model includes an image feature extraction layer, a feature fusion layer and a target detection layer, wherein: the text feature extraction layer, based on the attention mechanism, performs feature extraction on the input aircraft image to be identified to obtain text features; the text recognition layer, based on the logo detection result obtained by the target detection model, performs text recognition on the text features to obtain logo recognition results; the image feature extraction layer, performs feature extraction on the input aircraft image to be identified to obtain image features; the feature fusion layer, based on the feature pyramid, performs multimodal feature fusion on the text features and the image features to obtain pyramid fusion features; the target detection layer, performs target detection on the fused features to obtain logo detection results.

[0064] Specifically, the text feature extraction layer is used to: extract features from the input aircraft image to be identified to obtain feature vectors; perform position encoding on the feature vectors; determine the self-attention vector between the feature vector of each position and the feature vectors of other positions based on the multi-head attention mechanism and position encoding, and obtain the context representation vector of the feature vector corresponding to each position; use the feedforward neural network to map the context representation vector of the feature vector corresponding to each position into vector space to obtain the encoding feature; decode the encoding feature to obtain the text feature. It should be added that a residual connection is set between the output and input of each multi-head attention and feedforward neural network, and layer normalization is performed to improve the stability of the model.

[0065] Furthermore, the text feature extraction layer is also used to: based on the multi-head attention mechanism, for each attention head, perform a linear transformation on the position-encoded feature vector to obtain the corresponding query vector, key vector and value vector; obtain the attention weight according to the query vector and the key vector; obtain the output vector of the corresponding attention head based on the attention weight and the value vector; concatenate and linearly transform the output vectors of multiple attention heads to obtain the self-attention vector between the feature vector of each position of the multi-head self-attention output and the feature vectors at other positions.

[0066] In addition, the text recognition layer is used to: based on the identification detection result currently obtained by the target detection model, select the text features in the corresponding detection box and the identification recognition result obtained previously, perform probability distribution prediction on the next character, obtain the corresponding predicted probability distribution, and select the character with the highest predicted probability distribution as the corresponding identification recognition result; loop the probability distribution prediction for the next character and select the character with the highest predicted probability distribution as the corresponding identification recognition result until the sum of the character lengths of all the identification recognition results obtained meets the preset length; or, based on the identification detection result currently obtained by the target detection model, select the text features in the corresponding detection box, and classify the selected text features to map each text feature to the corresponding character category to obtain the identification recognition result.

[0067] In an optional embodiment, the system further includes: a text correction module, which, after obtaining the identification recognition result, corrects the identification recognition result based on preset character legality rules and preset format rules to remove repeated characters, correct character or format errors, etc., to avoid repeated recognition of the same character, improve the standardization and usability of the result, and make the final result more accurate.

[0068] In an optional embodiment, the image feature extraction layer is used to extract features from the input aircraft image to be identified layer by layer from low layers to high layers to obtain image features.

[0069] Correspondingly, the feature fusion layer is used to: perform convolution and pooling on the image features extracted from the first layer from high layer to low layer to obtain the first feature; upsample the image features extracted from the first layer, and perform feature fusion of the obtained upsampled features with the image features extracted from the second layer to obtain the corresponding fused features; wherein the first layer is higher than the second layer; upsample the fused features, and perform feature fusion of the obtained upsampled features with the image features extracted from the corresponding lower layer to obtain the corresponding fused features; repeatedly upsample the fused features, and perform feature fusion of the obtained upsampled features with the image features extracted from the corresponding lower layer, until the image features extracted from the lowest layer are fused with the upsampled features from the corresponding higher layer, and obtain the pyramid fused features according to the first feature and all the corresponding fused features obtained.

[0070] In an optional embodiment, the system further includes: a first screening module, which, after performing target detection and text recognition using a target detection model and a text recognition model based on the image of the aircraft to be identified to obtain a logo detection result and a logo recognition result, determines a confidence threshold based on the confidence of the logo detection result and the context information of the logo recognition result, and uses the confidence threshold to screen the logo detection result.

[0071] In an optional embodiment, the system further includes: a second screening module, which uses non-maximum suppression to remove redundant detection boxes in the logo detection results after performing target detection and text recognition using a target detection model and a text recognition model based on the image of the aircraft to be identified and obtaining the logo detection results and logo recognition results, thereby improving the accuracy of the detection results.

[0072] In an optional embodiment, the system further includes: a result correction module, which, after performing target detection and text recognition based on the aircraft image to be identified using the target detection model and the text recognition model to obtain the identification detection result and the identification recognition result, inputs the identification detection result and the identification recognition result corresponding to each aircraft image to be identified within the target time into the result correction model to obtain the correction result output by the result correction model; wherein the result correction model is obtained based on the aircraft image training data within the historical time period and the corresponding identification detection true value and identification recognition true value training.

[0073] In an optional embodiment, the system further includes: a training data acquisition module, which acquires aircraft image training data and identification detection labels and identification recognition labels corresponding to the aircraft image training data before performing target detection and text recognition based on the aircraft image to be recognized using the target detection model and the text recognition model to obtain identification detection results and identification recognition results; a collaborative training module, which uses the aircraft image training data as input data for training the target detection model to be trained, uses the identification detection labels corresponding to the aircraft image training data as labels for training the target detection model to be trained, and uses the aircraft image training data as input data for training the text recognition model to be trained, uses the identification recognition labels corresponding to the aircraft image training data as labels for training the text recognition model to be trained, and uses the collaborative mechanism to train the target detection model to be trained and the text recognition model to be trained in parallel.

[0074] Specifically, the training data acquisition module includes: a data acquisition unit, which collects aircraft images under different lighting (daytime, nighttime) and weather conditions based on diversified data sources to obtain aircraft image training data; a labeling unit, which labels the collected aircraft images to obtain logo detection labels and logo recognition labels.

[0075] In an optional embodiment, the system further includes: an image enhancement module, which performs image enhancement on the aircraft image training data after obtaining the aircraft image training data, such as histogram equalization, CLAHE (contrast limited adaptive histogram equalization), etc., and automatically adjusts according to the illumination and contrast of the image.

[0076] In an optional embodiment, the system further includes: a filtering module, which uses a preset filtering technology to perform denoising on the image of the aircraft to be identified after acquiring the image of the aircraft to be identified, so as to remove high-frequency noise in the image; and uses an adaptive sharpening technology to sharpen the denoised image of the aircraft to be identified, so as to enhance image details and improve image clarity.

[0077] In addition, the system also includes: a color correction module, after obtaining the image of the aircraft to be identified, performs white balance adjustment and color correction on the image to be identified, so as to ensure the consistency of image color and reduce the impact of lighting changes.

[0078] Furthermore, the system also includes: a data enhancement module, which performs data enhancement on the aircraft image training data based on a preset data enhancement strategy after the aircraft image training data; wherein the preset data enhancement strategy includes image distortion, text occlusion and noise addition, etc., and / or the preset data enhancement strategy also includes random cropping, scaling, etc., to improve the robustness of the model.

[0079] In an optional embodiment, in the process of parallel training of the target detection model to be trained and the text recognition model to be trained, model pruning and quantization techniques can also be used to reduce the number of model parameters, improve the inference speed, and adapt to the deployment requirements of resource-constrained devices.

[0080] In summary, the embodiments of the present invention utilize the target detection model and the text recognition model in a coordinated manner through the identification recognition module to achieve efficient collaborative processing of multiple tasks, and effectively combine the image features extracted by the target detection model and the text features extracted by the text recognition model to accurately detect the target representation, and utilize the identification detection results obtained by the detection to identify the text features, thereby utilizing the identification detection results to provide contextual information for text recognition, helping text recognition to better understand the position and meaning of the text in the image, further improving the robustness of the model in different scenarios, and improving the overall recognition accuracy.

[0081] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute an intelligent aviation sign recognition method based on visual multimodality, which includes: obtaining an aircraft image to be identified; based on the aircraft image to be identified, collaboratively using a target detection model and a text recognition model to perform target detection and text recognition to obtain a sign detection result and a sign recognition result; wherein the target detection model and the text recognition model are obtained by training based on aircraft image training data and sign detection labels and sign recognition labels corresponding to the aircraft image training data; the text recognition model is used to extract features from the input aircraft image to be identified, and perform text recognition on the extracted text features based on the sign detection results obtained by the target detection model to obtain a sign recognition result; the target detection model is used to extract features from the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain a sign detection result.

[0082] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0083] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the intelligent aviation identification method based on visual multimodality provided by the above-mentioned methods, and the method includes: obtaining an aircraft image to be identified; based on the aircraft image to be identified, collaboratively using a target detection model and a text recognition model to perform target detection and text recognition to obtain a logo detection result and a logo recognition result; wherein the target detection model and the text recognition model are obtained by training based on aircraft image training data and logo detection labels and logo recognition labels corresponding to the aircraft image training data; the text recognition model is used to extract features of the input aircraft image to be identified, and perform text recognition on the extracted text features based on the logo detection results obtained by the target detection model to obtain a logo recognition result; the target detection model is used to extract features of the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain a logo detection result.

[0084] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the visual multimodal intelligent aviation sign recognition method provided by the above-mentioned methods, the method comprising: obtaining an aircraft image to be identified; based on the aircraft image to be identified, collaboratively using a target detection model and a text recognition model to perform target detection and text recognition to obtain a sign detection result and a sign recognition result; wherein the target detection model and the text recognition model are obtained by training based on aircraft image training data and sign detection labels and sign recognition labels corresponding to the aircraft image training data; the text recognition model is used to extract features from the input aircraft image to be identified, and perform text recognition on the extracted text features based on the sign detection results obtained by the target detection model to obtain a sign recognition result; the target detection model is used to extract features from the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and text features to perform target detection on the fused features to obtain a sign detection result.

[0085] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent aviation sign recognition method based on visual multimodality, characterized in that: include: Obtain an image of an aircraft to be identified; According to the aircraft image to be identified, the target detection model and the text recognition model are used in collaboration to perform target detection and text recognition to obtain a logo detection result and a logo recognition result; wherein the target detection model and the text recognition model are trained based on aircraft image training data and logo detection labels and logo recognition labels corresponding to the aircraft image training data; The text recognition model is used to extract features of the input aircraft image to be recognized, and perform text recognition on the extracted text features based on the logo detection result obtained by the target detection model to obtain a logo recognition result; The target detection model is used to extract features from the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and the text features, so as to perform target detection on the fused features and obtain identification detection results.

2. The method for intelligent aviation sign recognition based on visual multimodality according to claim 1 is characterized in that: The text recognition model includes a text feature extraction layer and a text recognition layer, and the target detection model includes an image feature extraction layer, a feature fusion layer and a target detection layer, wherein: The text feature extraction layer extracts features of the input aircraft image to be identified based on the attention mechanism to obtain text features; The text recognition layer performs text recognition on the text features based on the logo detection result obtained by the target detection model to obtain a logo recognition result; The image feature extraction layer extracts features from the input aircraft image to be identified to obtain image features; The feature fusion layer performs multimodal feature fusion on the text feature and the image feature based on the feature pyramid to obtain a pyramid fusion feature; The target detection layer performs target detection on the fused features to obtain a mark detection result.

3. The method for intelligent aviation sign recognition based on visual multimodality according to claim 2 is characterized in that: Based on the attention mechanism, feature extraction is performed on the input aircraft image to be identified to obtain text features, including: Extract features from the input aircraft image to be identified to obtain a feature vector; Position encoding the feature vector; Based on the multi-head attention mechanism and the position encoding, a self-attention vector between the feature vector of each position and the feature vectors of other positions is determined to obtain a context representation vector corresponding to the feature vector of each position; Using a feedforward neural network, mapping the context representation vector of the feature vector corresponding to each position into a vector space to obtain a coding feature; The encoded features are decoded to obtain text features.

4. The method for intelligent aviation sign recognition based on visual multimodality according to claim 3 is characterized in that: Based on the identification detection result obtained by the target detection model, text recognition is performed on the text feature to obtain an identification recognition result, including: Based on the mark detection result currently obtained by the target detection model, select the text features in the corresponding detection box and the mark recognition result obtained once before, perform probability distribution prediction on the next character, obtain the corresponding predicted probability distribution, and select the character with the highest predicted probability distribution as the corresponding mark recognition result; Cycle to predict the probability distribution of the next character and select the character with the highest predicted probability distribution as the corresponding identification result until the sum of the character lengths of all identification results obtained meets the preset length; or, Based on the logo detection result currently obtained by the target detection model, text features in the corresponding detection box are selected, and the selected text features are classified to map each text feature to a corresponding character category to obtain a logo recognition result.

5. The method for intelligent aviation sign recognition based on visual multimodality according to claim 2 is characterized in that: Perform feature extraction on the input aircraft image to be identified to obtain image features, including: From the input aircraft image to be identified, features are extracted layer by layer from low level to high level to obtain image features; Based on the feature pyramid, the text feature and the image feature are subjected to multimodal feature fusion to obtain a pyramid fusion feature, including: From high layers to low layers, the image features extracted in the first layer are convolved and pooled to obtain the first features; Upsampling the image features extracted by the first layer, and fusing the obtained upsampled features with the image features extracted by the second layer to obtain corresponding fused features; wherein the level of the first layer is higher than that of the second layer; Upsampling the fused features, and fusing the obtained upsampled features with the image features extracted at a corresponding lower layer to obtain corresponding fused features; The fused features are repeatedly upsampled, and the obtained upsampled features are fused with the image features extracted from the corresponding lower layer until the image features extracted from the lowest layer are fused with the upsampled features from the corresponding higher layer. A pyramid fused feature is obtained based on the first feature and all the corresponding fused features obtained.

6. The method for intelligent aviation sign recognition based on visual multimodality according to claim 1 is characterized in that: After performing target detection and text recognition using a target detection model and a text recognition model according to the aircraft image to be recognized, and obtaining a mark detection result and a mark recognition result, the method includes: Determining a confidence threshold according to the confidence of the identification detection result and the context information of the identification recognition result, and screening the identification detection result using the confidence threshold; and / or, Using non-maximum suppression to remove redundant detection frames in the identification detection result; and / or, Based on preset character legality rules and preset format rules, the identification result is corrected; and / or, The identification detection results and identification recognition results corresponding to each aircraft image to be identified within the target time are input into the result correction model to obtain the correction result output by the result correction model; wherein the result correction model is trained based on the aircraft image training data within the historical time period and the corresponding identification detection true value and identification recognition true value.

7. The method for intelligent aviation sign recognition based on visual multimodality according to claim 1 is characterized in that: Before performing target detection and text recognition using a target detection model and a text recognition model according to the aircraft image to be recognized and obtaining a mark detection result and a mark recognition result, the method includes: Acquire aircraft image training data and identification detection labels and identification recognition labels corresponding to the aircraft image training data; The aircraft image training data is used as input data for training the target detection model to be trained, and the identification detection labels corresponding to the aircraft image training data are used as labels for training the target detection model to be trained. The aircraft image training data is used as input data for training the text recognition model to be trained, and the identification recognition labels corresponding to the aircraft image training data are used as labels for training the text recognition model to be trained. The target detection model to be trained and the text recognition model to be trained are trained in parallel using a collaborative mechanism.

8. An intelligent aviation sign recognition system based on visual multimodality, characterized in that: include: A data acquisition module, which acquires the image of the aircraft to be identified; A logo recognition module, which performs target detection and text recognition by collaboratively using a target detection model and a text recognition model according to the aircraft image to be recognized, and obtains a logo detection result and a logo recognition result; wherein the target detection model and the text recognition model are obtained by training based on aircraft image training data and logo detection labels and logo recognition labels corresponding to the aircraft image training data; The text recognition model is used to extract features of the input aircraft image to be recognized, and perform text recognition based on the extracted text features to obtain a logo recognition result; The target detection model is used to extract features from the input aircraft image to be identified, and perform multimodal feature fusion based on the extracted image features and the text features, so as to perform target detection on the fused features and obtain identification detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the intelligent aviation sign recognition method based on visual multimodality as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent aviation sign recognition method based on visual multimodality as described in any one of claims 1 to 7 is implemented.

Images