Image-text information identification method and device, computer equipment and storage medium
By acquiring and fusing text and image feature vectors of graphic and text information, using cross attention mechanism and classifier, the problem of difficulty in identifying multimodal fake news in the existing technology is solved, and the accurate classification and recognition of graphic and text information is achieved. , improves the accuracy and robustness of identification.
Patent Information
- Application Number
- CN202411403851.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2024-12-06
AI Technical Summary
The existing fake news detection methods mainly focus on text analysis, and it is difficult to effectively identify and classify multimodal fake news that blends images and text, resulting in difficulty in identifying and classifying them.
By obtaining the text and image content of the target graphic and text information, inputting the NLP model and CV model respectively to generate feature vectors, and using the cross attention mechanism to fuse the feature vectors, and finally the classifier generates information recognition results to achieve accurate classification of graphic and text information and identification.
It improves the accuracy and robustness of multi-modal fake news, can more comprehensively represent the semantic content and visual characteristics of graphic and text information, and enhances the reliability and accuracy of classification and recognition.
Smart Images

Figure CN119091457A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a method, device, computer equipment and computer-readable storage medium for identifying graphic information. Background Art
[0002] At present, with the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech), and image and text information recognition technology is no exception. For example, in the compliance review scenario of financial institutions (such as banks), financial institutions often need to review a large number of documents, such as contracts, loan documents, insurance policies, financial statements, identity certificates, etc. These documents usually contain rich text information and image content (such as signatures, seals, company logos, etc.). Through image and text information recognition technology, financial institutions can realize the automation and intelligence of compliance review, improve business efficiency and customer satisfaction, and reduce operating costs and compliance risks. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on image and text information recognition technology.
[0003] At the same time, with the explosive growth of social media, the speed and breadth of information dissemination have reached unprecedented levels. However, the dark side of this phenomenon is the rapid spread of false news, which not only disrupts people's lives and causes public panic, but also affects public opinion, manipulates public attention, and undermines the credibility of social media platforms. In the current digital media environment, the fragmentation of information allows uncontrolled false information to spread rapidly. The speed, scope, depth and breadth of false news far exceed those of real news, mainly because lies are often more attractive and easy to resonate on social media. In particular, multimodal false news that combines images and texts is more likely to attract the audience's attention than pure text news due to its attractive image content, making it more difficult to identify and contain. At present, existing false news detection methods mainly focus on text analysis, but with the increase of multimodal false news that combines images and texts, the effectiveness of these methods is limited, making false detection and classification more difficult.
[0004] In summary, how to provide a method, device, computer equipment and computer-readable storage medium for image and text information recognition that can achieve accurate classification and recognition of target image and text information is a problem that urgently needs to be solved by those skilled in the art. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a method, apparatus, computer device and computer-readable storage medium for identifying graphic information, aiming to solve the problem of how to achieve accurate classification and identification of target graphic information.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] In a first aspect, the present invention provides a method for identifying graphic information, which comprises:
[0008] Acquire target image and text information, and extract information from the target image and text information to obtain text content and image content therein;
[0009] Inputting the text content into the NLP model to generate a text feature vector of the target graphic information;
[0010] Inputting the image content into the CV model to generate an image feature vector of the target image and text information;
[0011] Based on the cross attention mechanism, a cross attention operation is performed on the text feature vector and the image feature vector to generate a fused feature vector of the target image and text information;
[0012] Based on the fused feature vector, a classifier is used to generate an information recognition result of the target graphic information.
[0013] In a second aspect, the present invention provides a device for recognizing graphic information, comprising:
[0014] An information extraction module is used to obtain target image and text information, and extract information from the target image and text information to obtain text content and image content therein;
[0015] A text content input module, used to input the text content into the NLP model to generate a text feature vector of the target graphic information;
[0016] An image content input module, used to input the image content into the CV model to generate an image feature vector of the target image and text information;
[0017] An operation execution module, configured to perform a cross attention operation on the text feature vector and the image feature vector based on a cross attention mechanism to generate a fused feature vector of the target image and text information;
[0018] The result generation module is used to generate the information recognition result of the target graphic information by using a classifier based on the fused feature vector.
[0019] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned graphic information recognition method when executing the computer program.
[0020] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned graphic information recognition method.
[0021] Compared with the prior art, the present invention provides a method, device, computer equipment and computer-readable storage medium for identifying graphic information, wherein target graphic information is acquired and information is extracted from the target graphic information to obtain text content and image content therein; the text content is input into an NLP model to generate a text feature vector of the target graphic information; the image content is input into a CV model to generate an image feature vector of the target graphic information; based on a cross-attention mechanism, a cross-attention operation is performed on the text feature vector and the image feature vector to generate a fused feature vector of the target graphic information; based on the fused feature vector, a classifier is used to generate an information recognition result of the target graphic information; thereby, the present invention can achieve accurate classification and recognition of target graphic information. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 A schematic diagram of an application environment of a method for recognizing graphic information provided by an embodiment of the present invention.
[0024] Figure 2 A schematic flow chart of a method for identifying graphic information provided by an embodiment of the present invention.
[0025] Figure 3 A schematic diagram of a program module of a device for identifying graphic information provided by an embodiment of the present invention.
[0026] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention.
[0027] Figure 5 Another structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0030] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0031] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0032] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0033] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0034] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0035] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.
[0036] An embodiment of the present invention provides a method for recognizing graphic information, which can be applied in the following aspects: Figure 1 In the application environment shown, the client and the server communicate through the network. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0037] See also Figure 2 An embodiment of the present invention provides a method for identifying graphic information, wherein the method comprises the following steps:
[0038] S100, obtaining target graphic information, and extracting information from the target graphic information to obtain text content and image content therein;
[0039] S200, inputting the text content into an NLP model to generate a text feature vector of the target graphic information;
[0040] S300, inputting the image content into a CV model to generate an image feature vector of the target image and text information;
[0041] S400, based on the cross attention mechanism, performing a cross attention operation on the text feature vector and the image feature vector to generate a fusion feature vector of the target image and text information;
[0042] S500: Based on the fused feature vector, a classifier is used to generate an information recognition result of the target graphic information.
[0043] In specific implementation, the graphic information recognition method of this embodiment realizes accurate classification and recognition of target graphic information through a carefully designed multimodal feature extraction and fusion strategy; first, in step S100, by extracting information from the target graphic information, the text content and image content therein are obtained respectively, thereby ensuring the quality and integrity of the data for subsequent processing; then, in steps S200 and S300, the NLP model and CV model are used to conduct in-depth analysis of the text content and image content respectively, and generate feature vectors that can represent the key information of each modality, which enhances the understanding and representation capabilities of different types of data; step S400 is the core of the method of this embodiment. By applying the cross-attention mechanism, it is possible not only to identify the relationship between text and image content, but also to strengthen the focus on the most critical part of the graphic information, thereby generating a A fused feature vector that integrates text and image information is generated. This fused feature vector can more comprehensively represent the semantic content and visual features of the target image and text information, thereby improving the accuracy and robustness of classification and recognition; finally, in step S500, the fused feature vector is processed by a classifier to generate a final classification result of the target image and text information. The classification method based on multimodal fusion features in this embodiment can more accurately identify and classify image and text information compared to a single modality classification method, because the method in this embodiment comprehensively considers the text semantics and visual content of the image and text information, thereby improving the reliability and accuracy of classification and recognition; in addition, the method in this embodiment can also be widely used in various scenarios that require image and text information processing, such as financial document review, social media content supervision, advertising content authenticity verification, etc., and has high practical value and broad application prospects.
[0044] Among them, NLP (Natural Language Processing) models are a class of algorithms used to understand and generate natural language text. They can handle tasks such as text classification, sentiment analysis, machine translation, question-answering systems, and text summarization. These models are usually based on complex neural network architectures, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), or Transformers, especially BERT (Bidirectional Encoder Representations from Transformers) and its derivative models, as well as DeBERTa (an improved BERT model), which capture the complexity and contextual relationships of language by learning deep semantic representations of text data.
[0045] Among them, CV (computer vision) models focus on extracting information from images or multi-dimensional data and performing tasks such as image classification, target detection, image segmentation, and pattern recognition; these models are usually based on convolutional neural networks (CNNs), which can learn local to global features of images through hierarchical structures; modern CV models may include architectures such as ResNet, Inception, or Swin Transformer, which can be trained to recognize and understand different objects and concepts in visual scenes, thereby achieving high performance in various visual recognition tasks.
[0046] Furthermore, in one embodiment, the method for identifying graphic information, wherein the step S100 of acquiring target graphic information and extracting information from the target graphic information to obtain text content and image content therein, includes:
[0047] Get the graphic and text information input by the target user;
[0048] Preprocessing the graphic information to obtain target graphic information;
[0049] Information extraction is performed on the target graphic information to obtain text content and image content therein.
[0050] In specific implementation, this embodiment obtains the graphic information input by the target user and then preprocesses it to obtain the target graphic information to improve the availability and quality of the data, and finally effectively extracts the text content and image content in the target graphic information, thereby providing structured data input for further analysis, understanding and utilization of the target graphic information; this process ensures the clarity and consistency of the information, and lays a solid foundation for subsequent multimodal processing tasks, such as feature extraction, content analysis or automated decision support.
[0051] The specific implementation process of the steps in this embodiment is roughly as follows:
[0052] 1. Obtaining graphic information:
[0053] Receiving text and image information uploaded by the target user through the user interface, which may include text files, pictures, PDF documents or other formats containing text and images;
[0054] 2. Format verification and conversion:
[0055] Verify whether the format of the uploaded graphic and text information is supported. If not, convert it to a processable format, such as converting images to JPG or PNG, and converting documents to a text-readable format;
[0056] 3. Preprocessing of graphic information:
[0057] Improve image quality by reducing noise, adjusting brightness and contrast, cropping irrelevant parts, and scaling to standard size.
[0058] Clean the text content, including removing irrelevant symbols, standardizing character encoding, and dividing lines and paragraphs to improve the readability of the text and the accuracy of the analysis;
[0059] 4. Text content extraction:
[0060] If the target graphic information contains text in the image, use OCR technology to perform text recognition on the image and convert the text in the image into machine-readable characters;
[0061] If the text in the target graphic information exists in the form of a file, the file content is read directly;
[0062] 5. Image content extraction:
[0063] Extract image features, which may include visual information such as color, shape, and texture;
[0064] Perform necessary image processing operations on the image, such as feature detection, edge extraction, etc., to facilitate subsequent analysis;
[0065] Through the above process, this embodiment can ensure that the graphic information input by the target user is effectively processed and converted to obtain relevant text content and image content, so as to facilitate subsequent feature extraction, content analysis or automated decision support and other applications.
[0066] Furthermore, in one embodiment, the method for identifying graphic information, wherein the step S200 of inputting the text content into an NLP model to generate a text feature vector of the target graphic information, comprises:
[0067] Pre-built NLP models;
[0068] Inputting the text content into the NLP model to generate a first initial feature vector;
[0069] Feature selection is performed on the first initial feature vector to obtain a text feature vector of the target graphic information.
[0070] In specific implementation, this embodiment processes the input text content through a pre-built NLP model, thereby realizing the conversion from raw text to high-quality feature representation; first, the processed text content is received by the NLP model and an initial feature vector containing rich language information is generated; then, through the feature selection process, the most representative and discriminative features are screened out from the initial feature vector, and finally an optimized text feature vector is obtained, which captures the core attributes of the text content and provides more accurate and effective input for subsequent graphic information analysis, classification or other machine learning tasks; this process significantly improves the level of automation of text processing and enhances the ability to understand and classify text information.
[0071] The specific implementation process of the steps in this embodiment is roughly as follows:
[0072] 1. Build NLP model:
[0073] Choose a suitable pre-trained NLP model, such as BERT, DeBERTa, GPT or its variants, and fine-tune it according to task requirements;
[0074] 2. Text content input NLP model:
[0075] Tokenize the extracted text content and convert it into an input format acceptable to the NLP model;
[0076] The processed text content is input into the NLP model, and the NLP model encodes the text content through its trained network structure;
[0077] 3. Generate the first initial eigenvector:
[0078] The NLP model outputs an encoded representation of the text content, usually a high-dimensional feature vector that contains the semantic information of the text content;
[0079] This feature vector is the output of the last layer of the NLP model or the layer selected by a specific strategy. It can be a time step of the hidden state or an aggregation of time steps.
[0080] 4. Feature Selection:
[0081] Analyze the first initial feature vector and use feature selection techniques to select the features that are most helpful for subsequent tasks;
[0082] Feature selection can be achieved in many ways, such as model-based feature importance scoring, statistical methods, or using automatic feature selection algorithms;
[0083] 5. Generate text feature vector:
[0084] Integrate the filtered features to form the final text feature vector;
[0085] This text feature vector is a compact representation of the text part of the image information, which can be used for subsequent classification, clustering or other machine learning tasks;
[0086] Through the above process, this embodiment can ensure that useful information is extracted from the text content and converted into a text feature vector that can effectively support the decision-making process.
[0087] Furthermore, in one embodiment, the method for recognizing graphic information, wherein the step S300 of inputting the image content into a CV model to generate an image feature vector of the target graphic information, comprises:
[0088] Pre-built CV models;
[0089] Inputting the image content into the CV model to generate a second initial feature vector;
[0090] Feature selection is performed on the second initial feature vector to obtain an image feature vector of the target graphic information.
[0091] In specific implementation, this embodiment uses a pre-built CV model to conduct an in-depth analysis of the image content, thereby realizing the conversion from raw image data to high-quality visual feature representation; first, the CV model receives the processed image content and generates an initial feature vector containing rich visual information; then, through a feature selection process, the most representative and discriminative features are screened out from the initial feature vector, and finally an optimized image feature vector is obtained; this image feature vector captures the core visual attributes of the image content, and provides more accurate and effective input for subsequent graphic information analysis, classification or other image processing tasks; this process significantly improves the level of automation of image processing and enhances the ability to understand and classify image information.
[0092] The specific implementation process of the steps in this embodiment is roughly as follows:
[0093] 1. Build CV model:
[0094] Choose a suitable pre-trained CV model, such as ResNet, Inception, Swin Transformer, or VGG, and fine-tune it according to task requirements;
[0095] 2. Image content input CV model:
[0096] Perform necessary preprocessing on the image content, such as resizing to match the input dimension of the CV model, normalizing pixel values, etc.
[0097] Input the preprocessed image content into the CV model;
[0098] 3. Generate the second initial eigenvector:
[0099] The CV model analyzes the input image content and generates feature maps in its internal convolutional layer;
[0100] Select the output of a certain layer in the CV model (usually the layer close to the output) as the second initial feature vector of the image content;
[0101] 4. Feature Selection:
[0102] Analyzing the second initial feature vector, feature-based importance assessment, principal component analysis (PCA) or other feature selection techniques may be used to identify the most informative features;
[0103] Depending on the results of feature selection, it may be necessary to remove some unimportant features or combine multiple features to form more representative features;
[0104] 5. Generate image feature vector:
[0105] Integrate the filtered features to form the final image feature vector;
[0106] This image feature vector is a compact representation of the image part of the image and text information, which can be used for subsequent image and text information fusion, classification or other machine learning tasks;
[0107] Through the above process, this embodiment can ensure that useful visual information is extracted from the image content and converted into an image feature vector that can effectively support the decision-making process.
[0108] Furthermore, in one embodiment, the method for recognizing graphic information, wherein the step S400, based on the cross attention mechanism, performs a cross attention operation on the text feature vector and the image feature vector to generate a fused feature vector of the target graphic information, comprises:
[0109] Performing an adjustment operation on the text feature vector and the image feature vector to ensure that the two are consistent in format and dimension;
[0110] Using the text feature vector as a query and the image feature vector as a key and a value, a cross-attention mechanism is used to calculate the similarity between the key in the text feature vector and the image feature vector, and generate an attention weight;
[0111] Based on the attention weights, the values in the image feature vectors are weighted summed to generate a fused feature vector of the target image and text information.
[0112] Furthermore, in the method for recognizing graphic information, the step of adjusting the text feature vector and the image feature vector to ensure that the two are consistent in format and dimension includes:
[0113] Analyzing the formats of the text feature vector and the image feature vector, and when the format analysis result shows that the formats of the two are inconsistent, adaptively adjusting the formats of the text feature vector and the image feature vector to ensure that the formats of the text feature vector and the image feature vector are consistent;
[0114] Analyze the dimensions of the text feature vector and the image feature vector. When the dimension analysis result shows that the dimensions of the two are inconsistent, adaptively adjust the dimensions of the text feature vector and the image feature vector to ensure that the text feature vector and the image feature vector are consistent in dimension.
[0115] In specific implementation, this embodiment achieves effective multimodal feature fusion by ensuring the consistency of the format and dimension of the text feature vector and the image feature vector; first, by analyzing and adjusting the format and dimension of the feature vector, the differences between data of different modalities are eliminated, laying the foundation for the application of the cross-attention mechanism; then, using the cross-attention mechanism, the similarity between text and image features can be calculated, and corresponding attention weights can be generated, which highlight the parts of the image features that are most relevant to the text; finally, by weighted summation of image features, a fused feature vector that combines text and image information is generated, which can more comprehensively characterize the semantic content of graphic information, thereby improving the accuracy and efficiency of multimodal data processing, and providing a rich and consistent feature representation for subsequent analysis, classification or retrieval tasks.
[0116] The specific implementation process of the steps in this embodiment is roughly as follows:
[0117] 1. Feature vector format and dimension analysis:
[0118] Check the format of text feature vectors and image feature vectors, including data type, structure, and value range;
[0119] Confirm the dimensionality information of both, i.e. the length of the feature vector and the scale of each feature in the vector;
[0120] 2. Feature vector format adjustment:
[0121] If the formats of text feature vectors and image feature vectors are inconsistent, such as inconsistent data types or large differences in numerical ranges, use appropriate data preprocessing techniques to adjust them, such as normalization or standardization;
[0122] Ensure that the adjusted text feature vector and image feature vector are consistent in numerical representation and structure to facilitate subsequent processing;
[0123] 3. Feature vector dimension adjustment:
[0124] If the dimensions of the text feature vector and the image feature vector are inconsistent, use techniques such as feature extraction, dimensionality reduction, or dimensionality expansion to adjust them;
[0125] For example, principal component analysis (PCA) can be used to reduce the dimensionality of high-dimensional features, or the dimension of low-dimensional feature vectors can be expanded by adding additional features;
[0126] 4. Perform cross-attention operation:
[0127] Take the text feature vector as the query and split the image feature vector into keys and values;
[0128] Using the cross attention mechanism, the similarity between the keys in the text feature vector and the image feature vector is calculated to generate attention weights;
[0129] 5. Generate fused feature vector:
[0130] According to the calculated attention weights, the values in the image feature vector are weighted summed;
[0131] The result of the weighted summation is combined with the text feature vector to form a fused feature vector, which integrates the semantic information of the text and the image;
[0132] Through the above process, this embodiment can ensure that the text feature vector and the image feature vector are consistent in format and dimension before the cross-attention calculation, and can generate a fused feature vector that integrates information from the two modalities, providing rich feature representation for subsequent classification, retrieval or other machine learning tasks.
[0133] Furthermore, in one embodiment, the method for identifying graphic information, wherein the step S500, based on the fused feature vector, using a classifier to generate the information identification result of the target graphic information, includes:
[0134] Pre-built classifiers;
[0135] Inputting the fused feature vector into the classifier to generate an information recognition result of the target graphic information;
[0136] The information identification result is formatted, and the formatted information identification result is output or displayed.
[0137] In specific implementation, this embodiment realizes efficient classification processing of fused feature vectors of graphic and text information through a pre-built classifier; first, the fused feature vector obtained from the multimodal feature extraction and fusion process is input into the classifier, and the text and image information integrated by the fused feature vector is used to make accurate category predictions; then, the information recognition results are formatted, such as converted into easy-to-understand text labels or visual displays, to ensure the user-friendliness and readability of the results; finally, the formatted information recognition results are output or displayed, so that users can clearly understand the classification of graphic and text information, thereby realizing rapid identification of graphic and text information and effective decision support in practical applications; this process significantly improves the automation and intelligence level of classification tasks, and provides strong technical support for the management and utilization of graphic and text information.
[0138] The specific implementation process of the steps in this embodiment is roughly as follows:
[0139] 1. Build a classifier:
[0140] Select a suitable machine learning algorithm, such as support vector machine (SVM), random forest, neural network, etc., to build a classification model;
[0141] If it is a neural network, design a suitable network architecture, including input layer, hidden layer and output layer;
[0142] 2. Training the classifier:
[0143] Use the labeled training data set to train the classifier, with the fused feature vector as input and the classification label as output;
[0144] The classifier parameters are adjusted by an optimization algorithm to minimize the difference between the predicted output and the actual label;
[0145] 3. Input fusion feature vector:
[0146] Input the fused feature vector obtained in the image and text information recognition process into the trained classifier;
[0147] The classifier analyzes and calculates the fused feature vector to determine the classification of the image and text information;
[0148] 4. Generate information recognition results:
[0149] The classifier outputs a prediction, which may be a class label or a probability distribution over classes.
[0150] For multi-class classification problems, the label corresponding to the maximum probability is selected as the prediction result;
[0151] 5.Format information recognition results:
[0152] Format the information recognition results as needed, such as converting them into JSON format, XML format or direct text description;
[0153] Design a format that is easy to understand and display so that the end user or system can clearly interpret the information recognition results;
[0154] 6. Output or display information recognition results:
[0155] Output the formatted information recognition results to an external system, storage system or directly display them on the user interface;
[0156] If necessary, additional information such as result explanation, confidence score or recommended actions can be further provided;
[0157] Through the above process, this embodiment can ensure that the classification process of graphic information is not only accurate and efficient, but also the results are easy to understand and use, meeting the needs of different application scenarios.
[0158] It can be seen from the above method embodiments that the method for recognizing graphic information provided by the present invention includes: obtaining target graphic information, and extracting information from the target graphic information to obtain text content and image content therein; inputting the text content into the NLP model to generate a text feature vector of the target graphic information; inputting the image content into the CV model to generate an image feature vector of the target graphic information; based on the cross-attention mechanism, performing a cross-attention operation on the text feature vector and the image feature vector to generate a fused feature vector of the target graphic information; based on the fused feature vector, using a classifier to generate an information recognition result of the target graphic information. In this way, accurate classification and recognition of target graphic information can be achieved through the method of the present invention.
[0159] It can be understood that the graphic information recognition method provided by the embodiment of the present invention can be applied to graphic information recognition scenarios related to the field of financial technology. Specifically, the graphic information recognition function can be integrated into the compliance review scenarios of financial institutions (such as banks). Financial institutions often need to review a large number of documents, such as contracts, loan documents, insurance policies, financial statements, identity certificates, etc. These documents usually contain rich text information and image content (such as signatures, seals, company logos, etc.). Through the graphic information recognition method provided by the embodiment of the present invention, financial institutions can realize the automation and intelligence of compliance reviews, improve business efficiency and customer satisfaction, and reduce operating costs and compliance risks.
[0160] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.
[0161] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a device for identifying graphic information, wherein the device comprises:
[0162] The information extraction module 11 is used to obtain target image and text information, and extract information from the target image and text information to obtain text content and image content therein;
[0163] A text content input module 12, used to input the text content into the NLP model to generate a text feature vector of the target graphic information;
[0164] An image content input module 13, used to input the image content into a CV model to generate an image feature vector of the target image and text information;
[0165] An operation execution module 14 is used to perform a cross attention operation on the text feature vector and the image feature vector based on a cross attention mechanism to generate a fusion feature vector of the target image and text information;
[0166] The result generating module 15 is used to generate the information recognition result of the target graphic information by using a classifier based on the fused feature vector.
[0167] Furthermore, in one embodiment, in the apparatus for identifying graphic information, the information extraction module 11 is specifically used for:
[0168] Get the graphic and text information input by the target user;
[0169] Preprocessing the graphic information to obtain target graphic information;
[0170] Information extraction is performed on the target graphic information to obtain text content and image content therein.
[0171] Furthermore, in one embodiment, in the apparatus for recognizing graphic information, the text content input module 12 is specifically used for:
[0172] Pre-built NLP models;
[0173] Inputting the text content into the NLP model to generate a first initial feature vector;
[0174] Feature selection is performed on the first initial feature vector to obtain a text feature vector of the target graphic information.
[0175] Furthermore, in one embodiment, in the apparatus for identifying graphic information, the image content input module 13 is specifically used for:
[0176] Pre-built CV models;
[0177] Inputting the image content into the CV model to generate a second initial feature vector;
[0178] Feature selection is performed on the second initial feature vector to obtain an image feature vector of the target graphic information.
[0179] Furthermore, in one embodiment, in the apparatus for identifying graphic information, the operation execution module 14 is specifically used to:
[0180] Performing an adjustment operation on the text feature vector and the image feature vector to ensure that the two are consistent in format and dimension;
[0181] Using the text feature vector as a query and the image feature vector as a key and a value, a cross-attention mechanism is used to calculate the similarity between the key in the text feature vector and the image feature vector, and generate an attention weight;
[0182] Based on the attention weights, the values in the image feature vectors are weighted summed to generate a fused feature vector of the target image and text information.
[0183] Furthermore, in the apparatus for identifying graphic information, the step of performing an adjustment operation on the text feature vector and the image feature vector to ensure that the two are consistent in format and dimension includes:
[0184] Analyzing the formats of the text feature vector and the image feature vector, and when the format analysis result shows that the formats of the two are inconsistent, adaptively adjusting the formats of the text feature vector and the image feature vector to ensure that the formats of the text feature vector and the image feature vector are consistent;
[0185] Analyze the dimensions of the text feature vector and the image feature vector. When the dimension analysis result shows that the dimensions of the two are inconsistent, adaptively adjust the dimensions of the text feature vector and the image feature vector to ensure that the text feature vector and the image feature vector are consistent in dimension.
[0186] Furthermore, in one embodiment, in the apparatus for identifying graphic information, the result generating module 15 is specifically used for:
[0187] Pre-built classifiers;
[0188] Inputting the fused feature vector into the classifier to generate an information recognition result of the target graphic information;
[0189] The information identification result is formatted, and the formatted information identification result is output or displayed.
[0190] It should be noted that in the embodiment of the device of the present invention, the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the aforementioned method embodiment part and will not be repeated here.
[0191] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the service side of the graphic information recognition method in any of the above method embodiments are implemented.
[0192] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps of the client side of the graphic information recognition method in any of the above method embodiments are implemented.
[0193] Those skilled in the art will understand that Figure 4 and Figure 5 The structural schematic diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0194] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.
[0195] Among them, the memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of a computer device, and in other embodiments, it can also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory can also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory can also be used to temporarily store data that has been output or is to be output.
[0196] Based on the above method embodiments, another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the graphic information recognition method in any of the above method embodiments is implemented. The computer-readable storage medium may be non-volatile or volatile.
[0197] It should be noted that the above-mentioned functions or steps that can be implemented by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be found in the relevant descriptions in the aforementioned method embodiments. To avoid repetition, they will not be described one by one here.
[0198] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). The disclosed memory components or memories of the operating environments described herein are intended to comprise one or more of these and / or any other suitable types of memory.
[0199] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, in the embodiment of the device of the present invention, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0200] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0201] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0202] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0203] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A method for identifying graphic information, characterized in that: include: Acquire target image and text information, and extract information from the target image and text information to obtain text content and image content therein; Inputting the text content into the NLP model to generate a text feature vector of the target graphic information; Inputting the image content into the CV model to generate an image feature vector of the target image and text information; Based on the cross attention mechanism, a cross attention operation is performed on the text feature vector and the image feature vector to generate a fused feature vector of the target image and text information; Based on the fused feature vector, a classifier is used to generate an information recognition result of the target graphic information.
2. The method for recognizing graphic information according to claim 1, characterized in that: The step of acquiring target graphic information and extracting information from the target graphic information to obtain text content and image content therein includes: Get the graphic and text information input by the target user; Preprocessing the graphic information to obtain target graphic information; Information extraction is performed on the target graphic information to obtain text content and image content therein.
3. The method for recognizing graphic information according to claim 1, characterized in that: The step of inputting the text content into the NLP model to generate a text feature vector of the target graphic information includes: Pre-built NLP models; Inputting the text content into the NLP model to generate a first initial feature vector; Feature selection is performed on the first initial feature vector to obtain a text feature vector of the target graphic information.
4. The method for recognizing graphic information according to claim 1, characterized in that: The step of inputting the image content into a CV model to generate an image feature vector of the target image and text information includes: Pre-built CV models; Inputting the image content into the CV model to generate a second initial feature vector; Feature selection is performed on the second initial feature vector to obtain an image feature vector of the target graphic information.
5. The method for recognizing graphic information according to claim 1, characterized in that: The method of performing a cross attention operation on the text feature vector and the image feature vector based on the cross attention mechanism to generate a fused feature vector of the target image and text information includes: Performing an adjustment operation on the text feature vector and the image feature vector to ensure that the two are consistent in format and dimension; Using the text feature vector as a query and the image feature vector as a key and a value, a cross-attention mechanism is used to calculate the similarity between the key in the text feature vector and the image feature vector, and generate an attention weight; Based on the attention weights, the values in the image feature vectors are weighted summed to generate a fused feature vector of the target image and text information.
6. The method for recognizing graphic information according to claim 5, characterized in that: The performing an adjustment operation on the text feature vector and the image feature vector to ensure that the two are consistent in format and dimension includes: Analyzing the formats of the text feature vector and the image feature vector, and when the format analysis result shows that the formats of the two are inconsistent, adaptively adjusting the formats of the text feature vector and the image feature vector to ensure that the text feature vector and the image feature vector are consistent in format; Analyze the dimensions of the text feature vector and the image feature vector. When the dimension analysis result shows that the dimensions of the two are inconsistent, adaptively adjust the dimensions of the text feature vector and the image feature vector to ensure that the text feature vector and the image feature vector are consistent in dimension.
7. The method for recognizing graphic information according to any one of claims 1 to 6, characterized in that: The step of generating the information recognition result of the target graphic information by using a classifier based on the fused feature vector includes: Pre-built classifiers; Inputting the fused feature vector into the classifier to generate an information recognition result of the target graphic information; The information identification result is formatted, and the formatted information identification result is output or displayed.
8. A device for recognizing graphic information, characterized in that: include: An information extraction module is used to obtain target image and text information, and extract information from the target image and text information to obtain text content and image content therein; A text content input module, used to input the text content into the NLP model to generate a text feature vector of the target graphic information; An image content input module, used to input the image content into the CV model to generate an image feature vector of the target image and text information; An operation execution module, configured to perform a cross attention operation on the text feature vector and the image feature vector based on a cross attention mechanism to generate a fused feature vector of the target image and text information; The result generation module is used to generate the information recognition result of the target graphic information by using a classifier based on the fused feature vector.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for recognizing graphic and text information as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for recognizing graphic and text information as described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Fault diagnosis system control method and device, and storage medium
CN120541500A
Digital archive automatic quality inspection method and system based on image recognition
CN120635922A
Artificial intelligence generated content identification method and related device
CN121010838A
