Multi-modal image content intelligent recognition method and device and medium

Abstract

The application belongs to the technical field of artificial intelligence and multi-modal data processing, and discloses a multi-modal image content intelligent identification method, device and medium. The method comprises the following steps: collecting business images and supporting business text metadata, completing text structured disassembly and image grid block; encoding text unit semantics and extracting visual features of image sub-blocks; calculating information entropy according to field types relying on a historical library, and performing dynamic semantic weighted normalization on text features; filtering out redundancy by using an information bottleneck filtering module, and extracting low-frequency key discriminant text features; adaptively aggregating image visual features, realizing cross-modal feature fusion through cross attention, and sending the recognition and labeling results output by a multi-modal recognition classifier. The application balances field information density, alleviates cross-modal information density mismatch, improves the classification accuracy of ambiguous business samples, and guarantees the reliable operation of upper-layer businesses.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal data processing technology, specifically relating to a method, device and medium for intelligent recognition of multimodal image content. Background Technology

[0002] In the practical business scenario of multimodal image content intelligent recognition on the digital technology innovation public service platform, the platform receives a large amount of business image resources uploaded by enterprises, institutions, and industry users. These include various types of image data such as scanned copies of technical certificates, screenshots of project applications, photos of technical contracts, and business images related to technical challenges. At the same time, it collects business text metadata such as technical fields, transaction attributes, and contract terms. Relying on cross-modal fine-grained complementary feature mining and processing, it establishes a correlation mapping relationship between local fine-grained features of images and semantic features of textual business, and completes intelligent image recognition and automatic classification and labeling. This provides recognition input support for upper-level business operations such as the release of platform technology and funding needs, the matching of challenges and recruitment, targeted customized cooperation, automatic generation of technology contracts, and digital asset transactions. In existing multimodal processing solutions for this type of business, the unstructured raw text metadata on the business side is typically standardized, breaking it down into various structured text units such as technical fields, transaction attributes, and contract terms. Semantic encoding is then performed on each structured text unit to obtain textual business semantic features. Simultaneously, the business image is processed through grid segmentation, extracting fine-grained local visual features corresponding to each image sub-block. Subsequently, cross-modal association operations are used to establish a mapping relationship between the two types of features, thereby achieving information fusion between the image modality and the business text modality. This is used to distinguish between a large number of ambiguous business samples within the platform that are visually similar but have completely different actual business meanings. However, in the real business data environment of the platform, the information entropy distribution among different types of structured text units is extremely uneven, exhibiting significant semantic sparsity. Some structured text units, such as technical field tags, are mostly high-frequency, repetitive, discrete words within a finite set, resulting in a low amount of effective business differentiation information carried by a single unit. Other structured text units, such as contract terms fields, belong to long-tail, low-frequency business text, containing a large number of business entities, numbers, values, and constraints, resulting in a high density of business information carried by a single unit. In contrast, after an image is divided into grids, the local fine-grained visual features output by each sub-block have a fixed vector dimension, and the overall information density among the sub-blocks is relatively uniform. This results in significant differences in information density among different business fields within the text metadata modality, while the information density of the image fine-grained features tends to be balanced across sub-blocks, creating an objective mismatch in cross-modal information density in business data. Existing conventional cross-modal association processing methods mostly directly feed all encoded text semantic features and local fine-grained image features into an attention mechanism to learn association weights, relying on the model to autonomously learn the association relationships between different features from the sample data.This approach has inherent flaws in platform business scenarios: the key discriminative information that can distinguish ambiguous business samples is often hidden in high-density, low-frequency, long-tail subtle textual differences such as contract terms, including subtle differences in contract numbers, transaction constraint values, and technical boundary descriptions. These key discriminative signals themselves have low frequency of occurrence. Conversely, low-density, high-frequency general business fields in the technical field have high frequency of occurrence and stronger feature signal energy. During cross-modal association operations, the semantic features corresponding to high-frequency general fields dominate, squeezing out and drowning out low-frequency, high-density key discriminative textual features, thus diluting the subtle business semantic signals that can distinguish ambiguous samples. Furthermore, within the platform business, such ambiguous samples with similar appearances but different business meanings are naturally small samples, and the number of similar samples available for model training is scarce. Relying solely on the attention mechanism to autonomously learn feature weights lacks sufficient sample data to support the model in learning the importance of low-frequency key textual features, making it impossible to spontaneously suppress high-frequency general features and amplify low-frequency discriminative features. Ultimately, this leads to the failure of feature fusion in the cross-modal fine-grained complementary feature mining stage. The mapping relationship between local fine-grained features of the image and the semantic features of the text shifts, and highly similar, ambiguous business samples still exhibit feature space overlap. This problem directly propagates to downstream business links, causing multimodal recognition to output incorrect classification and labeling results. This further triggers business failures such as errors in automatic technology contract generation, mismatch between supply and demand in the bidding process, and abnormal classification of digital assets, reducing the overall reliability of technology transactions and bidding cooperation within the digital technology innovation public service platform. Therefore, in the data preprocessing and cross-modal feature fusion stages, how to leverage the platform's prior business knowledge to construct a dynamic semantic weighted normalization or information bottleneck filtering mechanism that does not rely on a large number of labeled samples, automatically identify and amplify low-frequency key text features that can distinguish ambiguous business samples, suppress signal interference from high-frequency general structured text units, and overcome the fusion failure caused by the semantic sparsity of structured text metadata and the mismatch between the information density of fine-grained image features, has become a novel data processing technology problem urgently needing to be solved in the implementation of multimodal image content intelligent recognition technology in this type of digital technology innovation public service platform. Summary of the Invention

[0003] The purpose of this invention is to provide a method for intelligent recognition of multimodal image content, as well as a device for intelligent recognition of multimodal image content and a computer-readable storage medium, to solve the problems mentioned in the background art.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] A multimodal image content intelligent recognition method includes the following steps: collecting business image data and accompanying business text metadata; structurally decomposing the business text metadata to obtain multiple structured text units; dividing the business image data into grid blocks to obtain multiple image sub-blocks; semantically encoding each structured text unit to obtain a set of text semantic feature vectors; extracting visual features from each image sub-block to obtain a set of fine-grained visual feature vectors for local images; and, based on the full set of structured text unit data already stored in the platform's historical business database, calculating the information entropy value for each business field type according to the business field type, and performing dynamic semantic weighted normalization processing on each text semantic feature vector according to the information entropy value to obtain a weighted normalization result. A weighted normalized text feature vector set is generated; an information bottleneck filtering module is constructed to filter and amplify the weighted normalized text feature vector set, extract low-frequency key discrimination features, and output a text discrimination enhancement feature vector set; an adaptive aggregation processing is performed on the image local fine-grained visual feature vector set to output an image aggregated visual feature vector set; the text discrimination enhancement feature vector set and the image aggregated visual feature vector set are input into a cross-modal feature fusion module, and a cross-attention mechanism is used to establish the correlation mapping between image local fine-grained visual differences and text key discrimination semantic differences, outputting a cross-modal fused feature vector; the cross-modal fused feature vector is input into a multimodal recognition classifier to output the recognition, classification and annotation results of the business image.

[0006] Furthermore, the step of structurally decomposing the business text metadata to obtain multiple structured text units includes: identifying field boundaries of the business text metadata to determine the start and end positions of each business field; dividing the business text metadata into multiple text fragments according to the start and end positions; and classifying each text fragment into the corresponding business field type in a predefined business field classification system to generate the structured text unit.

[0007] Furthermore, the step of calculating the information entropy value for each business field type by grouping according to the business field type includes: grouping the full set of structured text unit data according to the business field type to obtain a subset of text units corresponding to each business field type; performing word frequency statistics on the values ​​in the subset of text units to calculate the probability of occurrence of each value; and determining the information entropy value for each business field type based on the probability of occurrence using information entropy calculation logic, wherein the information entropy value is used to characterize the degree of dispersion of the data distribution of the corresponding business field type.

[0008] Furthermore, the step of performing dynamic semantic weighted normalization processing on each text semantic feature vector based on the information entropy value includes: normalizing the information entropy value of each business field type to obtain a normalized semantic sparsity index for each business field type; calculating the dynamic semantic weight coefficient corresponding to each business field type using a monotonically increasing mapping function based on the normalized semantic sparsity index; and performing a weighted operation on each text semantic feature vector and the dynamic semantic weight coefficient of its respective business field type, followed by norm normalization processing to obtain the set of weighted normalized text feature vectors.

[0009] Furthermore, the construction of the information bottleneck filtering module, which filters and amplifies the weighted normalized text feature vector set, includes: constructing an information bottleneck filtering module comprising an encoder network, a bottleneck compression layer, and a decoder network; training the information bottleneck filtering module with the optimization objective of maximizing the mutual information between the bottleneck features and the business classification labels and minimizing the mutual information between the bottleneck features and the original input features; inputting the weighted normalized text feature vector set into the trained information bottleneck filtering module, filtering high-frequency general redundant information and retaining low-frequency key discrimination information through the bottleneck compression layer, and outputting the text discrimination enhanced feature vector set.

[0010] Furthermore, the adaptive aggregation processing of the image local fine-grained visual feature vector set to output an image aggregated visual feature vector set includes: constructing a spatial position relationship matrix based on the grid position index of each image sub-block in the original business image; and using the spatial position relationship matrix to perform context-related aggregation on the image local fine-grained visual feature vector set to generate an image aggregated visual feature vector set containing global spatial layout information.

[0011] Furthermore, the method of establishing a correlation mapping between fine-grained visual differences in local images and semantic differences in key text discrimination using a cross-attention mechanism includes: using the set of text discrimination enhancement feature vectors as query vectors, and using the set of aggregated visual feature vectors of the images as key vectors and value vectors; calculating the similarity score between the query vector and the key vector to generate a cross-modal attention weight matrix; using the cross-modal attention weight matrix to perform a weighted summation of the value vectors to achieve targeted enhancement of image visual features by text semantics, and outputting the cross-modal fusion feature vector.

[0012] Furthermore, the business image data includes scanned copies of technical certificates, screenshots of the project application system interface, or photos of the signed pages of technical contracts; the business text metadata includes technical field descriptions, transaction attribute fields, or contract terms.

[0013] A multimodal image content intelligent recognition device includes: a data acquisition and preprocessing module, used to acquire business image data and accompanying business text metadata, perform structured decomposition of the business text metadata to obtain multiple structured text units, and perform grid-based segmentation of the business image data to obtain multiple image sub-blocks; a feature encoding module, used to perform semantic encoding on the structured text units respectively to obtain a set of text semantic feature vectors, and to extract visual features from the image sub-blocks respectively to obtain a set of image local fine-grained visual feature vectors; and a dynamic weighting module, used to calculate the information entropy value of each business field type based on the full set of structured text unit data already stored in the platform's historical business database, and to perform dynamic semantic weighted normalization processing on each text semantic feature vector according to the information entropy value to obtain a weighted normalized value. The system comprises the following modules: a text feature vector set; a feature filtering module, used to construct an information bottleneck filtering module, which filters and amplifies the weighted normalized text feature vector set, extracts low-frequency key discrimination features, and outputs a text discrimination enhancement feature vector set; a visual aggregation module, used to perform adaptive aggregation processing on the image local fine-grained visual feature vector set, and outputs an image aggregated visual feature vector set; a cross-modal fusion module, used to input the text discrimination enhancement feature vector set and the image aggregated visual feature vector set into the cross-modal feature fusion module, using a cross-attention mechanism to establish a correlation mapping between image local fine-grained visual differences and text key discrimination semantic differences, and outputs a cross-modal fused feature vector; and a recognition and classification module, used to input the cross-modal fused feature vector into a multimodal recognition classifier, and output the recognition, classification, and annotation results of the business image.

[0014] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described intelligent multimodal image content recognition method.

[0015] This application also discloses an electronic device, including:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the multimodal image content intelligent recognition method of the present invention.

[0019] Beneficial Effects: This application calculates the information entropy value of each business field type based on the full data of the platform's historical business database, and performs dynamic semantic weighted normalization processing on the text semantic feature vector accordingly. This effectively solves the problem of cross-modal fusion failure caused by the huge difference in information density between different business field types in existing technologies. The information entropy value can objectively measure the dispersion of data distribution of each business field type, giving higher weight coefficients to long-tail low-frequency business fields with high information density and lower weight coefficients to high-frequency general business fields with low information density. This allows for the rebalancing of information density within the text modality before cross-modal association operations, avoiding the squeezing and overwhelming of low-frequency key discrimination feature signals by high-frequency general field feature signals. Building upon this foundation, this application constructs an information bottleneck filtering module. Its dual optimization objectives are maximizing the mutual information between bottleneck features and business classification labels while minimizing the mutual information between bottleneck features and original input features. During bottleneck compression, it actively filters high-frequency, general, and redundant information irrelevant to the classification task, preserving and enhancing key discriminative information hidden in subtle, low-frequency textual differences. This further improves the discriminative ability of text modal features. Furthermore, this mechanism relies on prior knowledge of the platform's business operations and does not depend on a large number of labeled samples for weight learning, overcoming the limitation of attention mechanisms' inability to autonomously learn feature weights under conditions of scarce ambiguous samples. Simultaneously, this application performs adaptive aggregation processing on fine-grained visual feature vectors of local images based on spatial relationships and visual similarity. This makes the information density distribution of image modal features more concentrated and uniform, achieving density adaptation with the text modal features after dynamic weighting and information bottleneck filtering, thus improving the objective data situation of information density mismatch between the two modalities. Ultimately, through a cross-modal cross-attention mechanism, an accurate correlation mapping can be established between the semantic differences in key text discrimination and the fine-grained visual differences in local images. The aliasing phenomenon of business ambiguity samples with highly similar visual appearances but completely different business meanings in the feature space is effectively alleviated, and the accuracy of classification and labeling results shows a significant upward trend. Consequently, the operational reliability of downstream business operations such as automatic generation of technology contracts, demand matching for bidding, and digital asset classification continues to improve. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the multimodal image content intelligent recognition method of the present invention;

[0021] Figure 2 This is a flowchart illustrating the calculation of business field information entropy and normalized semantic sparsity index in an embodiment of the present invention.

[0022] Figure 3 This is a flowchart of the dynamic semantic weighted normalization processing of text semantic features in an embodiment of the present invention;

[0023] Figure 4This is a flowchart of the information bottleneck filtering module in an embodiment of the present invention.

[0024] Figure 5 This is a flowchart of the adaptive aggregation processing of visual features of image sub-blocks in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention provides a method for intelligent recognition of multimodal image content, such as... Figure 1 As shown, the steps include:

[0027] The system collects business image data and accompanying business text metadata. It then performs structured decomposition of the business text metadata to obtain multiple structured text units, and divides the business image data into grid blocks to obtain multiple image sub-blocks. Semantic encoding is then performed on each structured text unit to obtain a set of text semantic feature vectors. Visual feature extraction is then performed on each image sub-block to obtain a set of fine-grained visual feature vectors for local images. Based on the full set of structured text unit data already stored in the platform's historical business database, the system calculates the information entropy value for each business field type, grouped by business field type. Dynamic semantic weighted normalization is then performed on each text semantic feature vector based on the information entropy value to obtain weighted normalized text feature vectors. The system first constructs a weighted normalized text feature vector set; then, it constructs an information bottleneck filtering module to filter and amplify the set of weighted normalized text feature vectors, extracting low-frequency key discrimination features and outputting a set of text discrimination enhancement feature vectors; it then performs adaptive aggregation processing on the set of local fine-grained visual feature vectors of the image, outputting a set of aggregated visual feature vectors of the image; finally, it inputs the set of text discrimination enhancement feature vectors and the set of aggregated visual feature vectors of the image into a cross-modal feature fusion module, using a cross-attention mechanism to establish a correlation mapping between local fine-grained visual differences in the image and key semantic differences in text discrimination, outputting a cross-modal fused feature vector; and finally, it inputs the cross-modal fused feature vector into a multimodal recognition classifier, outputting the recognition, classification, and annotation results of the business image.

[0028] This application provides a multimodal image content intelligent recognition method, which includes the following steps:

[0029] S101. Collect business image data and supporting business text metadata, perform structured decomposition of the business text metadata to obtain multiple structured text units, and perform grid-based segmentation of the business image data to obtain multiple image sub-blocks.

[0030] S102. Semantically encode each structured text unit to obtain a set of text semantic feature vectors, and extract visual features from each image sub-block to obtain a set of image local fine-grained visual feature vectors.

[0031] S103. Based on the full set of structured text unit data already stored in the platform's historical business database, calculate the information entropy value of each business field type according to the business field type, and perform dynamic semantic weighted normalization processing on each text semantic feature vector according to the information entropy value to obtain a set of weighted normalized text feature vectors.

[0032] S104. Construct an information bottleneck filtering module to filter and amplify the weighted normalized text feature vector set, extract low-frequency key discrimination features, and output a text discrimination enhancement feature vector set.

[0033] S105. Perform adaptive aggregation processing on the set of fine-grained visual feature vectors of the local image and output the set of aggregated visual feature vectors of the image.

[0034] S106. Input the text discrimination enhancement feature vector set and the image aggregated visual feature vector set into the cross-modal feature fusion module, and use the cross-attention mechanism to establish the correlation mapping between the local fine-grained visual differences of the image and the key discrimination semantic differences of the text, and output the cross-modal fusion feature vector.

[0035] S107. Input the cross-modal fusion feature vector into the multimodal recognition classifier and output the recognition, classification and annotation results of the service image.

[0036] In this embodiment, regarding step S101, the Digital Technology Innovation Public Service Platform continuously receives multimodal business data uploaded by enterprises, institutions, and industry users through the front-end interface. Specifically, each upload operation includes a set of business image files and business text metadata associated with those image files. The business image files are stored in digital image format and include scanned copies of technical certificates, screenshots of the project application system interface, photos of the signed pages of technical contracts, and related illustrations describing technical challenges. After receiving the business image files, the platform stores them in the platform's image resource database and assigns a unique image resource identifier number to each business image. The business text metadata associated with the business image files is either filled in by the user through a form on the platform's front-end during the upload process or automatically collected by the platform from related business systems. The business text metadata exists in unstructured or semi-structured text form and includes textual descriptions of the technical field, field information related to transaction attributes, and the specific content of contract terms.

[0037] Specifically, the platform performs standardized decomposition processing on the received unstructured raw business text metadata. A predefined business field classification system includes technology field tag types, transaction attribute field types, and contract clause content types. Technology field tag types cover the technology's domain, technology direction, and technology maturity description fields; transaction attribute field types cover transaction methods, pricing models, and cooperation periods; and contract clause content types cover contract number, transaction amount, technical boundary constraints, and delivery and acceptance standards description fields. The standardized decomposition process first identifies the field boundaries of the raw business text metadata. Using a pre-trained named entity recognition model and regular expression matching rules, the start and end positions of each business field in the raw text are identified. Then, the raw text is segmented into multiple text fragments according to the field boundaries. Finally, each text fragment is categorized into its corresponding business field type to form a structured text unit. Each structured text unit records its associated business image resource identifier number, business field type identifier, text unit sequence number, and text unit content.

[0038] For example, the business images are simultaneously processed by grid partitioning. A uniform grid partitioning strategy is used to divide each business image into several rows and columns of grids in the spatial dimension, with each grid corresponding to an image sub-block. Each image sub-block retains the pixel data of the original image and records its grid position index, including the row index and column index. Each image sub-block is stored in the platform's image sub-block database and associated with its corresponding business image resource identifier number. Through unified acquisition and structured preprocessing of multimodal business data, a standardized data input foundation is provided for subsequent cross-modal feature encoding and fusion.

[0039] In this embodiment, for step S102, a pre-trained text semantic encoding model based on the Transformer architecture is used for feature encoding of the text modality. Specifically, the text content of each structured text unit is fed into the pre-trained text semantic encoding model as input. The model performs multi-layer self-attention encoding calculations on the input text and outputs a fixed-dimensional text semantic feature vector at the classification label position of the last layer of the model. The dimension of this text semantic feature vector is determined according to the model configuration, and the vector encodes the deep business semantic information of the corresponding structured text unit. The above encoding process is performed on all structured text units one by one, outputting a set of text semantic feature vectors. Each vector in the set corresponds one-to-one with a structured text unit and carries the business field type identifier of that text unit.

[0040] Specifically, for image modality feature encoding, a pre-trained image visual feature encoding model based on a convolutional neural network architecture or a Vision Transformer architecture is employed. The pixel data of each image sub-block is fed as input into the pre-trained image visual feature encoding model. The model performs multi-layer convolution or self-attention feature extraction calculations on the input image sub-blocks, outputting a fixed-dimensional local fine-grained visual feature vector. It should be noted that the aforementioned pre-trained image visual feature encoding models based on convolutional neural network architectures or Vision Transformer architectures are mature existing technologies in this field. Their specific network structures, parameter configurations, and training processes can be directly selected or conventionally implemented by those skilled in the art according to actual needs, and are not the focus of this application. Therefore, this application will not elaborate further to ensure sufficient disclosure in the specification.

[0041] The dimension of this visual feature vector is determined according to the model configuration. The vector encodes fine-grained visual information such as local visual texture, spatial structure, and color distribution of the corresponding image sub-block. The above encoding process is performed on each of all image sub-blocks, outputting a set of fine-grained local visual feature vectors. Each vector in the set corresponds one-to-one with an image sub-block and carries the grid position index of that sub-block. By performing feature encoding on the text modality and the image modality respectively, a set of text semantic feature vectors and a set of fine-grained local visual feature vectors for subsequent cross-modal association operations are obtained.

[0042] In this embodiment, for step S103, the platform accesses all structured text unit data already stored in the historical business database, and groups the all structured text units according to the business field type identifier to obtain a subset of structured text units corresponding to each business field type. For example... Figure 2As shown, specifically, information entropy calculation is performed on a subset of structured text units corresponding to any business field type. First, word frequency statistics are performed on the values ​​of all structured text units in the subset to obtain the frequency distribution of different values ​​under that business field type. For discrete values, the frequency of each word is directly counted; for continuous or long text values, the text content is first segmented into words before the frequency of each segmented unit is counted.

[0043] Specifically, let the first In the subset of structured text units corresponding to the various business field types, the first... The different values ​​are Its frequency of occurrence is The total number of different values ​​for this business field type is Then the probability of that value occurring is... The calculation is as follows:

[0044] ;

[0045] The information entropy value of this business field type is calculated using the information entropy calculation formula, which is the negative sum of the product of the probability of occurrence of each value under this business field type and its logarithmic value. Information entropy value of various business field types The calculation is as follows:

[0046] ;

[0047] After calculation, the information entropy value corresponding to each business field type is obtained. For example, the information entropy value of the technical field tag type is usually low because the value content under this type is concentrated in a limited set of technical field terms, with a few high-frequency terms accounting for most of the occurrence frequency; the information entropy value of the contract clause content type is usually high because the value content under this type contains a large number of business entity names, numbers, values, and constraint descriptions, with a wide value distribution and a large proportion of long-tail low-frequency content.

[0048] Specifically, the information entropy values ​​of each business field type are subjected to min-max normalization, mapping the information entropy values ​​to a preset normalization interval to obtain the normalized semantic sparsity index for each business field type. Let H be the minimum information entropy value among all business field types. min The maximum value is H max Then the first Normalized semantic sparsity index for various business field types The calculation is as follows:

[0049] ;

[0050] The value of the normalized semantic sparsity index is positively correlated with the information entropy of the business field type. The larger the value, the higher the amount of effective distinguishing information carried by the business field type and the richer the semantics. The smaller the value, the sparser the semantics of the business field type and the weaker the distinguishing ability of a single text unit.

[0051] like Figure 3 As shown, specifically, for each text semantic feature vector, the business field type identifier it carries is read, and the corresponding normalized semantic sparsity index value for that business field type is searched in the normalized semantic sparsity index set. Based on the found normalized semantic sparsity index value, a dynamic semantic weight coefficient is calculated. The calculation of the dynamic semantic weight coefficient uses a monotonically increasing mapping function to ensure a positive correlation between the normalized semantic sparsity index value and the dynamic semantic weight coefficient. Let the minimum value of the dynamic semantic weight coefficient be w. min The maximum value is w max Then the first Dynamic semantic weight coefficients corresponding to different business field types The calculation is as follows:

[0052] w k =w min +(w max -w min )S k ;

[0053] Each text semantic feature vector is element-wise multiplied by its corresponding dynamic semantic weight coefficient scalar value to obtain a weighted text semantic feature vector. After weighting, text semantic feature vectors belonging to high-information-density business fields such as contract clause content type receive larger weight coefficients, and their feature signal intensity is amplified; text semantic feature vectors belonging to low-information-density business fields such as technical field tag type receive smaller weight coefficients, and their feature signal intensity is suppressed.

[0054] Specifically, vector normalization is performed on the fully weighted text semantic feature vectors using the L2 norm normalization method. Let the... The text semantic feature vectors are The dynamic semantic weight coefficient of its business field type is The weighted normalized text semantic feature vector The calculation is as follows:

[0055] ;

[0056] This process ensures that the normalized set of text semantic feature vectors participates in subsequent calculations under a unified feature scale. After this step, the information density differences between different business field types within the text modality are rebalanced, the signal interference of high-frequency, general, low-information-content text features in the subsequent cross-modal fusion process is suppressed in advance, and the signals of low-frequency, high-density, high-information-content text features acquire a relative strength within the text modality that matches their distinguishing importance.

[0057] In this embodiment, for step S104, an information bottleneck filtering module is constructed, which includes three sub-modules: an encoder network, a bottleneck compression layer, and a decoder network. For example... Figure 4 As shown, specifically, the encoder network consists of multiple fully connected layers that map the high-dimensional text semantic feature vectors of the input to a low-dimensional bottleneck feature space; the bottleneck compression layer sets an information compression bottleneck, with the bottleneck dimension lower than the dimension of the input text semantic feature vectors; the decoder network consists of multiple fully connected layers that map the low-dimensional compressed features in the bottleneck feature space back to the original dimension space.

[0058] Specifically, the training optimization objective function of the information bottleneck filtering module includes two mutual information constraint terms. The mutual information preservation term maximizes the mutual information between the bottleneck-compressed features and the business classification labels, ensuring that key discriminative information capable of distinguishing ambiguous business samples is preserved during compression. The mutual information compression term minimizes the mutual information between the bottleneck-compressed features and the original input text semantic features, forcing the model to discard general redundant information irrelevant to the business classification task during compression. These two constraints are balanced using Lagrange multipliers. During training, existing business image classification annotation data from the platform is used as supervision signals, and variational approximation methods are employed to estimate and optimize the mutual information preservation and compression terms. During training, the encoder network learns to extract and compress key discriminative information from the weighted text semantic feature vector into the bottleneck feature space, while discarding high-frequency general redundant information; the decoder network learns to restore the compressed discriminative information from the bottleneck feature space to the original dimensional space.

[0059] Specifically, during the inference phase, the weighted and normalized set of text semantic feature vectors is input into the trained information bottleneck filtering module. This module is then mapped to the bottleneck feature space via the encoder network. High-frequency general feature information is filtered out and discarded during bottleneck compression, while low-frequency key discriminative feature information is retained and restored to its original dimensionality space via the decoder network, outputting a set of text discriminative enhanced feature vectors. After the information bottleneck filtering process, low-frequency key discriminative signals relevant to the task of distinguishing ambiguous business samples are preserved and enhanced, while high-frequency general redundant signals unrelated to the discrimination task are compressed and suppressed, further improving the discriminative ability of text modal features.

[0060] In this embodiment, for step S105, an image feature adaptive aggregation module is constructed, which internally includes an aggregation weight calculation unit. For example... Figure 5 As shown, specifically, the aggregation weight calculation unit includes a spatial location encoding subunit and a visual similarity calculation subunit. The spatial location encoding subunit generates a corresponding location encoding vector based on the grid location index of each image sub-block. This location encoding vector encodes the spatial location information of the image sub-block in the original business image, enabling the subsequent aggregation process to perceive the spatial adjacency relationship between image sub-blocks. The visual similarity calculation subunit calculates the cosine similarity between the visual feature vectors of any two image sub-blocks, obtaining a visual similarity matrix between image sub-blocks. This matrix reflects the degree of correlation between each image sub-block at the visual content level.

[0061] Specifically, the aggregation weight calculation unit jointly calculates the positional encoding vector and the visual similarity matrix to generate an adaptive aggregation weight vector for each image sub-block. Each element of this aggregation weight vector represents the contribution weight of the corresponding image sub-block to the current image sub-block during the aggregation process. Image sub-blocks that are spatially adjacent and visually similar receive higher contribution weights, while image sub-blocks that are spatially dispersed and have significantly different visual content receive lower contribution weights. The visual feature vectors of each image sub-block are weighted and summed according to the aggregation weight vector to obtain the aggregated visual feature vector of each image sub-block. The aggregated visual feature vectors fuse the visual information of adjacent and similar sub-blocks, enhancing the visual features of local regions and smoothing and suppressing the interference of discrete noise sub-blocks. The aggregated visual feature vectors of all image sub-blocks constitute the image aggregated visual feature vector set. After adaptive aggregation processing, the information density distribution of the image modality visual features is more concentrated and uniform, forming a better match with the feature information density distribution of the text modality after dynamic weighting and information bottleneck filtering, providing density-adapted image feature input for subsequent cross-modal fusion.

[0062] In the business image recognition scenario addressed in this application, the semantic relationships between image sub-blocks are not only affected by spatial proximity and visual similarity, but also strongly constrained by the inherent layout structure of business documents. For example, although the contract number and amount fields are visually significantly different and spatially far apart, they belong to the same semantic unit. The above-mentioned aggregation mechanism relies solely on the joint weighting of position encoding and visual similarity, which is insufficient to capture such cross-regional structural semantic coupling relationships, resulting in the weakening of key discriminative information by local smoothing effects during the aggregation process. Therefore, it is necessary to introduce an aggregation strategy that can explicitly model the topological relationships of business layouts, so that the feature fusion process takes into account spatial neighborhood, visual consistency, and prior document structure. This will preserve fine-grained local features while strengthening the complete expression of cross-regional semantic units, ensuring that the aggregated visual features and the text features after dynamic weighting and information bottleneck processing are accurately aligned in terms of information density and semantic granularity, providing a more discriminative visual representation foundation for subsequent cross-modal fusion.

[0063] This application further proposes a fractional-order graph diffusion adaptive aggregation algorithm, which includes the following steps:

[0064] Input data includes: a set of fine-grained visual feature vectors of local images. ,in , The total number of grid blocks, For feature dimensions. Grid position index of each image sub-block. , For row index, For column indexes.

[0065] Algorithm applications include:

[0066] 1. Construct the adjacency matrix and degree matrix of the image sub-block graph structure.

[0067] For any two image sub-blocks and Calculate the joint weights of spatial Gaussian affinity and visual cosine similarity, and construct a weighted adjacency matrix. :

[0068] ;

[0069] in: This is a spatial scale parameter that controls the rate of decay of spatial distance; its value is adaptively determined by the total number of rows and columns of the grid. ( , (These represent the number of rows and columns of the grid, respectively). , For sub-blocks , The grid row and column coordinates; The inner product of the visual feature vectors; For vectors The L2 norm.

[0070] From the adjacency matrix Calculate the degree matrix ,in:

[0071] ;

[0072] Then construct the normalized graph Laplacian matrix. :

[0073] ;

[0074] in for Identity matrix. Let be a symmetric positive semi-definite matrix, whose eigenvalues ​​satisfy... .

[0075] 2. Construction of fractional-order graph diffusion kernels.

[0076] right Perform eigenvalue decomposition: ,in The matrix of orthogonal eigenvectors It is a diagonal matrix of eigenvalues.

[0077] Define fractional-order diffusion kernel matrix :

[0078] ;

[0079] Right now:

[0080] ;

[0081] in:

[0082] This is a fractional-order parameter, and it is the core control hyperparameter of this invention. When It degenerates into a classical integer-order graph heat kernel; when At that time, high-frequency signal components Compared to magnification, the diffusion process exhibits sub-diffusion characteristics, with information propagation confined to a local neighborhood, making it suitable for preserving the local fine-grained structure of image sub-blocks; when At this time, the low-frequency signal components decay faster, and the diffusion process exhibits super-diffusion characteristics, allowing information to propagate over long distances, making it suitable for aggregating spatially dispersed but semantically related sub-blocks; The diffusion time parameter controls the depth of information propagation on the graph. The larger the size, the wider the diffusion range; The first of the Graph Laplace matrix One eigenvalue; For the corresponding number 1 eigenvector.

[0083] 3. Adaptive determination of the fractional diffusion order α.

[0084] In order to make To adapt to the content complexity of images for different business applications, an adaptive calculation mechanism based on graph entropy is designed. First, the graph probability distribution is calculated:

[0085] ;

[0086] Calculate spectral entropy :

[0087] ;

[0088] Normalize the spectral entropy:

[0089] ;

[0090] Adaptive fractional order The calculation formula is:

[0091] ;

[0092] in:

[0093] Sensitivity coefficient (recommended value) ),control The strength of the response to graph structural complexity;

[0094] Normalized graph entropy reflects the degree of spectral dispersion of the graph structure.

[0095] Physical meaning: When the image content is uniform and the visual differences between sub-blocks are small (such as a scanned plain text document), the spectral energy is concentrated in the low frequency range. Smaller than average Diffusion tends to be localized and sub-diffusion, preserving fine-grained local textures; however, when image content is complex and contains multiple semantic regions (such as a contract page with mixed seals, signatures, and tables), the spectral energy is dispersed. Too big The diffusion tends to be cross-domain hyper-diffusion, enabling information exchange between semantically related sub-blocks over long distances.

[0096] 4. Perform fractional-order graph diffusion aggregation.

[0097] The original visual feature matrix Viewed as a row in the diagram The characteristic signals of each node are applied to a fractional-order diffusion kernel. A linear transformation yields the aggregated visual feature matrix:

[0098] ;

[0099] That is, each image sub-block The aggregated visual feature vector is:

[0100] ;

[0101] in For the fractional-order diffusion kernel matrix, the first... Line 1 Column elements represent sub-blocks Pair blocks Fractional diffusion contribution weights.

[0102] 5. Residual enhancement and normalized output.

[0103] To preserve the original local fine-grained information and prevent over-smoothing, residual connectivity and L2 normalization are introduced:

[0104] ;

[0105] in:

[0106] The residual mixing coefficient (recommended value) ), controlling the mixing ratio of aggregated features and original features;

[0107] Aggregate visual feature vectors for the final output image.

[0108] all Constructing a set of aggregated visual feature vectors for images The output is sent to the downstream cross-modal feature fusion module.

[0109] The output is a set of aggregated visual feature vectors of the image. , where each vector It is an image sub-block feature after fractional-order graph diffusion aggregation, which has completed the adaptive fusion of non-local semantic information, and the information density distribution is more concentrated and uniform, forming a density fit with the text modality feature.

[0110] This process elevates the aggregation process from empirical weighting to solving fractional partial differential equations with rigorous mathematical and physical significance. Without increasing the dependence on labeled samples, it provides an interpretable, controllable, and difficult-to-avoid technical barrier for image sub-block feature aggregation through spectral domain analysis.

[0111] In this embodiment, for step S106, a cross-modal feature fusion module is constructed. This module adopts a multi-layer cross-attention mechanism architecture, including a text-to-image cross-modal attention calculation unit, an image-to-text cross-modal attention calculation unit, and a feature concatenation and fusion layer. Specifically, in the text-to-image cross-modal attention calculation unit, the set of text discrimination enhancement feature vectors is used as the query vector input, and the set of image aggregated visual feature vectors is used as the key vector and value vector input. The scaling dot product attention weight between the query vector and the key vector is calculated to obtain the attention weight distribution matrix of text features to image features. This attention weight distribution matrix reflects the cross-modal association strength between each text semantic feature vector and each image visual feature vector. The attention weight distribution matrix is ​​applied to the value vectors for weighted summation to obtain the text-guided image enhancement feature vector set.

[0112] Specifically, in the cross-modal attention computation unit from image to text, the set of aggregated visual feature vectors of the image is used as the input query vector, and the set of text discriminative enhancement feature vectors is used as the input key and value vectors. The scaled dot product attention weights between the query vector and the key vector are calculated to obtain the attention weight distribution matrix of image features on text features. The attention weight distribution matrix is ​​applied to the value vectors and weighted to obtain the set of image-guided text enhancement feature vectors. The set of text-guided image enhancement feature vectors is concatenated element-wise with the set of image-guided text enhancement feature vectors to obtain the concatenated cross-modal joint feature vector. The concatenated cross-modal joint feature vector is fed into a fully connected mapping layer for dimensionality compression and nonlinear transformation, outputting the final cross-modal fusion feature vector. Since the text modal features have undergone dynamic semantic weighting based on the information entropy measurement results of the business field types, the feature signal intensity of high-frequency, general, low-information fields is suppressed, while the feature signal intensity of low-frequency, high-density, high-information fields is amplified. Furthermore, after information bottleneck filtering, general redundant information is further compressed, and key discriminant information is preserved and enhanced. Therefore, in the cross-modal cross-attention calculation process, text features belonging to key discriminant fields such as contract terms can obtain sufficient attention allocation in the attention weight distribution and are no longer squeezed and submerged by high-frequency, general field features such as technical field labels. An accurate cross-modal association mapping is established between the key discriminant signals hidden in the long-tail, low-frequency, subtle text differences in the text modality and the local fine-grained visual difference signals in the image modality, realizing the effective fusion of fine-grained complementary features.

[0113] In this embodiment, for step S107, a multimodal recognition classifier is constructed. This classifier includes a feature abstraction network and a classification output layer. Specifically, the feature abstraction network consists of multiple fully connected networks, with nonlinear activation functions and regularization layers set between layers to further abstract and map the input cross-modal fusion feature vector into higher dimensions. The classification output layer maps the output of the feature abstraction network to a predefined business classification label space, outputting the predicted probability distribution of each business category. The cross-modal fusion feature vector is input into the multimodal recognition classifier. After layer-by-layer calculation by the feature abstraction network, the classification output layer outputs the classification prediction result of the business image. The classification prediction result includes the business category label to which the business image belongs (e.g., technical certificate, project application, technical contract, technical problem), the business attribute label (e.g., technical field classification, transaction status classification), and the business status label (e.g., approval status, pending review status). The category with the highest predicted probability is taken as the final classification result.

[0114] Specifically, the classification prediction results are associated and bound with the corresponding business image data records. Classification labeling fields are written into the business image data records to generate business image data records with complete classification labeling information. The labeled business image data records are written to the platform's business database, and the labeling result notification is pushed to the upper-layer business system through the platform's data service interface. Upon receiving the labeling result notification, the upper-layer business system executes corresponding business logic processing based on the classification labeling information. The technology requirement publishing module pushes technology requirement information to the matching target user group based on business category tags and technology field classification tags; the challenge-and-response matching module performs precise matching and recommendation between technology suppliers and technology demanders based on the classification labeling information; the automatic technology contract generation module calls the corresponding contract template and fills in the business field content based on the classification labeling information; and the digital asset trading module performs digital asset classification, warehousing, and transaction matching based on the classification labeling information. Because the cross-modal fusion feature vector accurately encodes the correlation mapping relationship between fine-grained visual information of the image locality and key discriminative semantic information of the text, the multimodal recognition classifier can accurately distinguish ambiguous business samples that are visually similar but have completely different business meanings. The accuracy of the classification and labeling results is improved, and the operational reliability of downstream business links such as automatic generation of technical contracts, demand matching, and digital asset classification is guaranteed.

[0115] In some embodiments, the steps in the above method embodiments can be implemented by software, hardware, or a combination of both. When implemented in software, the computer program corresponding to the above method steps can be stored in a computer-readable storage medium and loaded and executed by the processor of an electronic device.

[0116] This application provides a multimodal image content intelligent recognition device. The device includes:

[0117] The data acquisition and preprocessing module is used to acquire business image data and supporting business text metadata, perform structured decomposition of the business text metadata to obtain multiple structured text units, and perform grid-based segmentation of the business image data to obtain multiple image sub-blocks.

[0118] The feature encoding module is used to perform semantic encoding on the structured text units respectively to obtain a set of text semantic feature vectors, and to extract visual features from the image sub-blocks respectively to obtain a set of fine-grained visual feature vectors of the local image.

[0119] The dynamic semantic weighted normalization module is used to calculate the information entropy value of each business field type based on the full amount of structured text unit data already stored in the platform's historical business database, grouped according to the business field type, and perform dynamic semantic weighted normalization processing on each text semantic feature vector according to the information entropy value to obtain a set of weighted normalized text feature vectors.

[0120] The information bottleneck filtering module is used to filter and amplify the weighted normalized text feature vector set, extract low-frequency key discrimination features, and output a text discrimination enhancement feature vector set.

[0121] The adaptive aggregation module is used to perform adaptive aggregation processing on the set of fine-grained local visual feature vectors of the image and output the set of aggregated visual feature vectors of the image.

[0122] The cross-modal feature fusion module is used to input the text discrimination enhancement feature vector set and the image aggregated visual feature vector set into the cross-modal feature fusion module, and to establish the correlation mapping between the local fine-grained visual differences of the image and the key discrimination semantic differences of the text using a cross-attention mechanism, and output the cross-modal fused feature vector;

[0123] The recognition and classification module is used to input the cross-modal fusion feature vector into the multimodal recognition classifier and output the recognition, classification and annotation results of the service image.

[0124] Optionally, the specific functional implementation of each module in the above device embodiments can be referred to the corresponding step description in the above method embodiments, and will not be repeated here.

[0125] This application provides an electronic device. The electronic device includes at least one processor and a memory communicatively connected to the processor. The memory stores instructions executable by the at least one processor, which, when executed by the processor, enables the processor to perform the multimodal image content intelligent recognition method described in any of the above-described method embodiments. The electronic device may further include an input / output interface and a communication unit. The input / output interface is used to receive business image data and accompanying business text metadata, and the communication unit is used to interact with the platform's historical business database and upper-layer business systems.

[0126] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the multimodal image content intelligent recognition method described in any of the above method embodiments. Specifically, the computer-readable storage medium may include, but is not limited to, a hard disk, an optical disk, flash memory, a read-only memory, a random access memory, etc.

[0127] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal image content intelligent recognition method described in any of the above method embodiments.

[0128] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal image content intelligent recognition method, characterized in that, Includes the following steps: The system collects business image data and accompanying business text metadata, decomposes the business text metadata into structured units, and divides the business image data into grid blocks to obtain multiple image sub-blocks. Semantic encoding is then performed on each structured text unit to obtain a set of text semantic feature vectors. Visual features are then extracted from each image sub-block to obtain a set of fine-grained visual feature vectors for local images. Based on the full set of structured text unit data already stored in the platform's historical business database, the information entropy value for each business field type is calculated, grouped according to the business field type. Dynamic semantic weighted normalization is then performed on each text semantic feature vector based on the information entropy value to obtain a set of weighted normalized text feature vectors. An information bottleneck filtering module is constructed to filter and amplify the weighted normalized text feature vector set, extract low-frequency key discrimination features, and output a text discrimination enhancement feature vector set. Adaptive aggregation processing is performed on the image local fine-grained visual feature vector set to output an image aggregated visual feature vector set. The text discrimination enhancement feature vector set and the image aggregated visual feature vector set are input into a cross-modal feature fusion module, which uses a cross-attention mechanism to establish a correlation mapping between image local fine-grained visual differences and text key discrimination semantic differences, outputting a cross-modal fused feature vector. The cross-modal fused feature vector is input into a multimodal recognition classifier to output the recognition, classification, and annotation results of the business image.

2. The multimodal image content intelligent recognition method according to claim 1, characterized in that, The step of structurally decomposing the business text metadata to obtain multiple structured text units includes: identifying field boundaries of the business text metadata to determine the start and end positions of each business field; dividing the business text metadata into multiple text fragments according to the start and end positions; and classifying each text fragment into the corresponding business field type in a predefined business field classification system to generate the structured text unit.

3. The multimodal image content intelligent recognition method according to claim 1, characterized in that, The step of calculating the information entropy value for each business field type by grouping according to the business field type includes: grouping the full set of structured text unit data according to the business field type to obtain a subset of text units corresponding to each business field type; performing word frequency statistics on the values ​​in the subset of text units to calculate the probability of occurrence of each value; and determining the information entropy value for each business field type based on the probability of occurrence using information entropy calculation logic, wherein the information entropy value is used to characterize the degree of dispersion of the data distribution of the corresponding business field type.

4. The multimodal image content intelligent recognition method according to claim 3, characterized in that, The step of performing dynamic semantic weighted normalization processing on each text semantic feature vector based on the information entropy value includes: normalizing the information entropy value of each business field type to obtain a normalized semantic sparsity index for each business field type; calculating the dynamic semantic weight coefficient corresponding to each business field type using a monotonically increasing mapping function based on the normalized semantic sparsity index; and performing a weighted operation on each text semantic feature vector and the dynamic semantic weight coefficient of its respective business field type, followed by norm normalization processing to obtain the set of weighted normalized text feature vectors.

5. The multimodal image content intelligent recognition method according to claim 1, characterized in that, The construction of the information bottleneck filtering module, which filters and amplifies the weighted normalized text feature vector set, includes: constructing an information bottleneck filtering module comprising an encoder network, a bottleneck compression layer, and a decoder network; training the information bottleneck filtering module with the optimization objective of maximizing the mutual information between the bottleneck features and the business classification labels and minimizing the mutual information between the bottleneck features and the original input features; inputting the weighted normalized text feature vector set into the trained information bottleneck filtering module, filtering high-frequency general redundant information and retaining low-frequency key discrimination information through the bottleneck compression layer, and outputting the text discrimination enhanced feature vector set.

6. The multimodal image content intelligent recognition method according to claim 1, characterized in that, The adaptive aggregation processing of the image local fine-grained visual feature vector set to output an image aggregated visual feature vector set includes: constructing a spatial position relationship matrix based on the grid position index of each image sub-block in the original business image; and using the spatial position relationship matrix to perform context-related aggregation on the image local fine-grained visual feature vector set to generate an image aggregated visual feature vector set containing global spatial layout information.

7. The multimodal image content intelligent recognition method according to claim 1, characterized in that, The method of establishing a correlation mapping between fine-grained visual differences in local images and key semantic differences in text using a cross-attention mechanism includes: using the set of text-based discriminative enhancement feature vectors as query vectors and the set of aggregated visual feature vectors in the images as key vectors and value vectors; calculating the similarity score between the query vector and the key vector to generate a cross-modal attention weight matrix; using the cross-modal attention weight matrix to perform a weighted summation of the value vectors to achieve targeted enhancement of image visual features by text semantics, and outputting the cross-modal fusion feature vector.

8. The multimodal image content intelligent recognition method according to any one of claims 1 to 7, characterized in that, The business image data includes scanned copies of technical certificates, screenshots of the project application system interface, or photos of the signed pages of technical contracts; the business text metadata includes technical field descriptions, transaction attribute fields, or contract terms.

9. A multimodal image content intelligent recognition device, characterized in that, include: The data acquisition and preprocessing module is used to acquire business image data and supporting business text metadata, perform structured decomposition of the business text metadata to obtain multiple structured text units, and perform grid-based segmentation of the business image data to obtain multiple image sub-blocks. The feature encoding module is used to perform semantic encoding on the structured text units respectively to obtain a set of text semantic feature vectors, and to extract visual features from the image sub-blocks respectively to obtain a set of image local fine-grained visual feature vectors; the dynamic weighting module is used to calculate the information entropy value of each business field type based on the full set of structured text unit data already stored in the platform's historical business database, and to perform dynamic semantic weighting and normalization processing on each text semantic feature vector according to the information entropy value to obtain a set of weighted normalized text feature vectors. The feature filtering module is used to construct an information bottleneck filtering module, which filters and amplifies the weighted normalized text feature vector set, extracts low-frequency key discrimination features, and outputs a text discrimination enhancement feature vector set; the visual aggregation module is used to perform adaptive aggregation processing on the image local fine-grained visual feature vector set, and outputs an image aggregated visual feature vector set; the cross-modal fusion module is used to input the text discrimination enhancement feature vector set and the image aggregated visual feature vector set into the cross-modal feature fusion module, and establishes the correlation mapping between image local fine-grained visual differences and text key discrimination semantic differences using a cross-attention mechanism, and outputs a cross-modal fused feature vector. The recognition and classification module is used to input the cross-modal fusion feature vector into the multimodal recognition classifier and output the recognition, classification and annotation results of the service image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal image content intelligent recognition method as described in any one of claims 1 to 8.

Images