Market illegal behavior detection method based on multi-modal image-text fusion

Abstract

The invention discloses a market violation behavior detection method based on multi-modal image-text fusion. The method comprises the following steps: acquiring text and image data of market business and market violation information as a basis; preprocessing the basic information to construct a market business violation detection data set; respectively extracting a text feature Ft and an image feature Fi of the data; performing feature self-enhancement and cross-modal alternative enhancement on the features of the text and the image through a feature enhancer of a mixed attention mechanism to obtain an enhanced text feature Ft'and an enhanced image feature Fi '; the original features and the enhanced features are input into a feature fusion device, feature fusion of Ft, Fi, Ft'and Fi 'is achieved through intermediate fusion and weighted fusion methods, and a final fusion feature Ff is obtained; and inputting the fusion features into a downstream illegal behavior detector for detection, and outputting a detection result. According to the method, association enhancement and fusion of intra-modal and inter-modal features can be realized by fully utilizing the relevance of texts and images, and illegal behaviors of information such as transactions, registration and records in market services can be more accurately judged and detected.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence multi-modal data processing. Specifically, it relates to a method for detecting market violation behaviors based on multi-modal text and image fusion. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, the detection of market violation behaviors has become an important part of maintaining market order and protecting consumers' rights and interests. Traditional methods for detecting market violation behaviors often rely on single-modal data, such as text, images, or videos. However, this single-modal processing method has many limitations, such as incomplete information acquisition and a single analysis perspective, making it difficult to comprehensively and accurately detect market violation behaviors.

[0003] In recent years, multi-modal fusion technology has gradually attracted the attention of researchers. Multi-modal fusion technology refers to the fusion of information from different modalities to obtain a more comprehensive and accurate information representation. In the fields of computer vision, natural language processing, etc., significant research results have been achieved in multi-modal fusion technology. For example, through the fusion of images and text, a deeper understanding of image content can be achieved; through the fusion of speech and text, more accurate speech recognition and natural language understanding can be realized.

[0004] However, applying multi-modal fusion technology to the field of market violation behavior detection still faces many challenges. First, the manifestations of market violation behaviors are diverse, involving information in multiple modalities such as text, images, and videos. How to effectively fuse this information is a difficult problem. Second, there is heterogeneity between information in different modalities. How to design a reasonable fusion algorithm so that information in different modalities can complement and enhance each other is another problem that needs to be solved. In addition, the detection of market violation behaviors also needs to consider requirements such as real-time performance and accuracy. How to improve the detection efficiency while ensuring the detection effect is also an important research direction.

[0005] Currently, some studies have attempted to apply multi-modal fusion technology to the detection of market violation behaviors. These studies mainly focus on feature extraction, fusion algorithms, and classifier design. However, these studies often only verify specific violation behaviors or specific data sets, lacking generality and generalization ability. In addition, the design of fusion algorithms in these studies is often relatively simple, without fully considering the interaction and complementary effects between information in different modalities. Summary of the Invention

[0006] According to the problems existing in the prior art, the present invention provides a method for detecting market violation behaviors based on multimodal text and image fusion. By comprehensively considering the information of both text and image modalities, making full use of the advantages of different modality information, performing multimodal feature enhancement and feature fusion, it realizes the comprehensive and accurate detection of market violation behaviors, and improves the accuracy, reliability, generality and generalization ability of market violation behavior detection.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A method for detecting market violation behaviors based on multimodal text and image fusion, comprising the following steps:

[0009] S10. Collect normal data and violation information in text format and image format in the market business;

[0010] S20. Preprocess the collected normal data and violation information to construct a market business violation detection data set;

[0011] S30. Use the ALBERT model to extract text feature F t from the market business violation detection data set, and use the ResNet-50 network model to extract image feature F i ;

[0012] S40. Through the feature enhancer with a hybrid attention mechanism, perform in-modal enhancement and cross-modal alternating enhancement on the extracted text feature F t and image feature F i to obtain enhanced text feature F t ' and enhanced image feature F i ';

[0013] S50. Input the text feature F t and image feature F i obtained in S30 and the enhanced text feature F t ' and enhanced image feature F i ' obtained in S40 into the feature fuser together, and realize the feature fusion of F t 、F i 、F t ' and F i ' through intermediate fusion and weighted fusion methods to obtain the final fused feature F f ;

[0014] S60. Input the fused feature F f into the downstream violation behavior detector for detection and output the detection result.

[0015] Specifically, the process of preprocessing the collected normal data and violation information to construct a market business violation detection dataset in step S20 is as follows:

[0016] S21. Organize the image-format data collected in the market business and number the image IDs;

[0017] S22. Organize the text-format data collected in the market business and store it in a dictionary data structure, storing it in key-value pair format, where the key corresponds to the image ID and the value corresponds to the text description;

[0018] S23. Divide the dataset into a training set, a validation set, and a test set according to the image IDs;

[0019] S24. Construct an annotations file to classify and label the violations of the image-format data. The annotation file includes the image ID, category ID, annotation ID, segmentation mask, and target box corresponding to each picture.

[0020] Specifically, in step S30, the ALBERT model is used to extract the text feature F from the market business violation detection dataset t , and the ResNet-50 network model is used to extract the image feature F from the market business violation detection dataset i The process is as follows:

[0021] S31. Select a pre-trained ALBERT model. After inputting the text description corresponding to a specific ID into the ALBERT model, the model outputs the embedded representation of the text, capturing the semantic information of the text;

[0022] S32. Load the ResNet-50 network model pre-trained on the large dataset ImageNet. After inputting the image corresponding to a specific ID into the ResNet-50 network model, the model extracts features layer by layer until the last convolutional layer or the global average pooling layer, and selects the output of a specific layer as the image feature;

[0023] S33. Adjust the feature dimensions by pooling the text feature and the image feature to extract key features.

[0024] Specifically, in step S40, through the feature enhancer with a hybrid attention mechanism, the extracted text feature F t and the image feature F i The process of performing in-feature-modal enhancement and cross-feature-modal alternating enhancement is as follows:

[0025] S41. For the text feature F t and the image feature F iFeature modality - internal enhancement is achieved through self - attention mechanisms. For text features, the multi - head self - attention mechanism of the Transformer architecture is used, and for image features, the self - attention mechanism of the Non - local block is used.

[0026] S42. The features obtained by feature modality - internal enhancement are passed through an alternating co - attention mechanism to achieve cross - modality alternating enhancement. The cross - modality attention mechanism of the Transformer architecture is used. The text features and image features after modality - internal enhancement are used to calculate weights, correlation degrees, and weighted sums through a cross - modality Transformer encoder to obtain enhanced text feature F t ’ and enhanced image feature F i ’.

[0027] Specifically, the process of achieving feature modality - internal enhancement in step S41 is as follows:

[0028] S411. For the original text feature F t The multi - head self - attention mechanism of the Transformer architecture is used. By performing embedding encoding on the text features of the input sequence, and then sending (Q, K, V) into the self - attention module of the encoder, a new encoded representation is calculated. The multi - head self - attention mechanism divides (Q, K, V) into N parts, each part is independently processed by the self - attention module, and finally the processing results of each block are fused, which can capture multiple relationships and subtle differences between input features.

[0029] S412. For the original image feature F i The self - attention mechanism of the Non - local non - local block is used to enhance the features of important information in the image. Through the non - local operation operator, it only requires fewer stacked layers to capture the long - range dependencies between the spatial positions of the image, and can more easily embed the calculation results into the current network structure on the premise of ensuring that the input scale and output scale are consistent.

[0030] Specifically, the process of achieving feature cross - modality alternating enhancement in step S42 is as follows:

[0031] S421. The cross - attention mechanism can capture the internal feature structure and feature correlation between modalities. By inputting the text features and image features after feature modality - internal enhancement into the alternating co - attention mechanism, and using a cross - modality Transformer encoder, the correlation degree between each region in the text features and image features is calculated, and this correlation degree is measured by attention weights.

[0032] S422. Using (Q i , K t , V t ) and (Q t , Ki ,V i The cross-modal three elements of (), respectively use text features and image features as Query queries, and then use image features and text features as Key elements that match the queries. Through the cross-modal attention mechanism, the attention weights are calculated, and then weighted summation operations are performed with the Value vectors obtained by matching the queries to obtain enhanced representation features.

[0033] Specifically, in step S50, the text feature F obtained in S30 t and the image feature F i and the enhanced text feature F obtained in S40 t ’ and the enhanced image feature F i ’ are jointly input into the feature fusion device, and the feature fusion process of F t , F i , F t ’ and F i ’ is as follows:

[0034] S51. Through the intermediate fusion idea, the enhanced text feature F obtained in S40 t ’ and the enhanced image feature F i ’ are input into multiple fully connected layers to encode the inter-modal correlation, and the intermediate fusion feature F m is fused;

[0035] S52. The text feature F obtained in S30 t and the image feature F i and the intermediate fusion feature F obtained in S51 m are weighted and fused. By calculating the similarity between the text feature F t and the image feature F i and the intermediate fusion feature F m , the attention weights are obtained, and then the weighted features are obtained through weighted fusion. Finally, the weighted features are fused through a splicing function to obtain the final fusion feature F f .

[0036] Specifically, the process of weighted fusion in step S52 is as follows:

[0037] S521. The text feature F t , the image feature F i and the intermediate fusion feature F m are weighted and fused. First, the feature similarity S between the intermediate fusion feature F m and the text feature F t and the feature similarity S between the intermediate fusion feature F t and the image feature F m and the image feature F iThe feature similarity S between i , and the calculation formula is:

[0038]

[0039] where m i represents the i-th eigenvalue in the intermediate fusion feature F m , t i represents the i-th eigenvalue in the text feature F t , m j represents the j-th eigenvalue in the intermediate fusion feature F m , t j represents the j-th eigenvalue in the image feature F i ;

[0040] S522. Then, use the SoftMax function to normalize the feature similarities S t and S i respectively to obtain the attention weights A t and A i , and the calculation formula is:

[0041] A t = SoftMax(S t )...(3)

[0042] A i = SoftMax(S i )...(4);

[0043] S523. Then, multiply the attention weights A t and A i with the intermediate fusion feature F m respectively to obtain the weighted fusion features F m t and F m i , and the calculation formula is:

[0044]

[0045] S524. Finally, use the Concat function to splice the weighted fusion features F m t and F m i to fuse and obtain the final fusion feature F f , and the calculation formula is:

[0046]

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] Based on obtaining the normal data and violation information of text and images in market operations, according to the extracted text and image features, perform intra-modal and inter-modal enhancement of the features, then fuse the original features and the enhanced features, and finally perform violation detection on the fused features. This method makes full use of the feature correlation of multi-modalities, can improve the accuracy and robustness of detection compared to single-modal text and images, and enhance the efficiency of market operation violation detection. The present invention greatly reduces the manual review of market operation violation data and behaviors, provides active detection for information such as market registration, transactions, and records; can perform feature enhancement from single-modal and multi-modal of text and images, fully mining the feature information of the data; can combine text and images for feature fusion, making full use of the inter-modal correlation to improve the detection accuracy. And the present invention also has a certain degree of generality and generalization ability, and can be applied to different types of market violation behavior detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of an embodiment of the present invention.

[0050] Figure 2 It is a flowchart of constructing a market operation violation detection data set in an embodiment of the present invention.

[0051] Figure 3 It is a flowchart of feature enhancement in an embodiment of the present invention.

[0052] Figure 4 It is a flowchart of feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The present invention will be further described below in conjunction with the drawings and embodiments. The implementation manners of the present invention include but are not limited to the following embodiments.

[0054] Embodiment

[0055] As Figures 1 to 4 shown, the market violation behavior detection method based on multi-modal text and image fusion includes the following steps:

[0056] S10. Collect the normal data and violation information in text format and image format in market operations.

[0057] S20. Preprocess the collected normal data and violation information to construct a market operation violation detection data set.

[0058] The specific process is as follows:

[0059] S21. Sort out the data in image format collected in market operations and number the image ids;

[0060] S22. Organize the text - formatted data collected in the market business and store it in a dictionary - form data structure, storing it in key - value pair format, where the key corresponds to the image id and the value corresponds to the text description;

[0061] S23. Divide the dataset into a training set, a validation set, and a test set according to the image id;

[0062] S24. Construct an annotations file to classify and label the violations of the image - formatted data. The annotation file includes the image id, category id, annotation id, segmentation mask, and target box corresponding to each picture.

[0063] S30. Use the ALBERT model to extract the text feature F from the market business violation detection dataset t , and use the ResNet - 50 network model to extract the image feature F from the market business violation detection dataset i .

[0064] The specific process is as follows:

[0065] S31. Select the pre - trained ALBERT model. After inputting the text description corresponding to a specific id into the ALBERT model, the model outputs the embedded representation of the text, capturing the semantic information of the text;

[0066] S32. Load the ResNet - 50 network model pre - trained on the large - scale dataset ImageNet. After inputting the image corresponding to a specific id into the ResNet - 50 network model, the model extracts features layer by layer until the last convolutional layer or the global average pooling layer, and selects the output of a specific layer as the image feature;

[0067] S33. Adjust the feature dimensions by pooling the text feature and the image feature to extract key features.

[0068] S40. Through the feature enhancer of the hybrid attention mechanism, perform in - modality enhancement of the feature and cross - modality alternating enhancement on the extracted text feature F t and the image feature F i to obtain the enhanced text feature F t ’ and the enhanced image feature F i ’.

[0069] The specific process is as follows:

[0070] S41. For the text feature F t and the image feature F iFeature modality internal enhancement is achieved through self-attention mechanism. For text features, the multi-head self-attention mechanism of the Transformer architecture is adopted, and for image features, the self-attention mechanism of the Non-local block is adopted. Among them,

[0071] S411. For the original text feature F t Adopt the multi-head self-attention mechanism of the Transformer architecture. By performing embedding encoding on the text features of the input sequence, and then sending (Q, K, V) into the self-attention module of the encoder, a new encoded representation is calculated. The multi-head self-attention mechanism divides (Q, K, V) into N parts, each part is independently processed by the self-attention module, and finally the processing results of each block are fused, which can capture multiple relationships and subtle differences between input features.

[0072] S412. For the original image feature F i Adopt the self-attention mechanism of the Non-local block to enhance the features of useful and important information in the image. Through the non-local operation operator, it only needs fewer stacked layers to capture the long-range dependencies between the spatial positions of the image, and can more easily embed the calculation results into the current network structure on the premise of ensuring that the input scale and output scale are consistent.

[0073] S42. The features obtained by feature modality internal enhancement are used through the alternating co-attention mechanism to achieve cross-modal alternating enhancement. Adopt the cross-modal attention mechanism of the Transformer architecture. The text features and image features enhanced within the modality are used to calculate weights, correlation degrees, and weighted sums through the cross-modal Transformer encoder to obtain the enhanced text feature F t ’ and the enhanced image feature F i ’. Among them,

[0074] S421. The cross-attention mechanism can capture the internal feature structure and feature correlation between modalities. By inputting the text features and image features enhanced within the feature modality into the alternating co-attention mechanism, and adopting the cross-modal Transformer encoder, the correlation degree between each region in the text features and image features is calculated, and this correlation degree is measured by the attention weight.

[0075] S422. Adopt (Q i , K t , V t ) and (Q t , K i , V iThe cross-modal three elements, respectively take text features and image features as Query queries, and then take image features and text features as Key elements matching the queries. Calculate the attention weights through the cross-modal attention mechanism, and then perform a weighted summation operation with the Value vector obtained by matching the queries to obtain the enhanced representation features.

[0076] S50. Input the text feature F obtained in S30 t and the image feature F i together with the enhanced text feature F obtained in S40 t ’ and the enhanced image feature F i ’ into the feature fuser, and implement the feature fusion of F t , F i , F t ’ and F i ’ through intermediate fusion and weighted fusion methods to obtain the final fused feature F f .

[0077] The specific process is as follows:

[0078] S51. Through the idea of intermediate fusion, input the enhanced text feature F obtained in S40 t ’ and the enhanced image feature F i ’ into multiple fully connected layers to encode the inter-modal correlation and fuse to obtain the intermediate fusion feature F m ;

[0079] S52. Perform weighted fusion on the text feature F obtained in S30 t and the image feature F i and the intermediate fusion feature F obtained in S51 m . Calculate the attention weights by computing the similarity between the text feature F t and the image feature F i and the intermediate fusion feature F m , then obtain the weighted feature through weighted fusion, and finally fuse the weighted feature through a concatenation function to obtain the final fused feature F f .

[0080] Among them, the process of performing weighted fusion is as follows:

[0081] S521. Perform weighted fusion on the text feature F t , the image feature F i and the intermediate fusion feature F m . First, calculate the feature similarity S m between the intermediate fusion feature F t and the text feature F t and the feature similarity between the intermediate fusion feature F m and the image feature Fi The feature similarity S between i , and the calculation formula is:

[0082]

[0083] Where m i represents the i-th eigenvalue in the intermediate fusion feature F m , t i represents the i-th eigenvalue in the text feature F t , m j represents the j-th eigenvalue in the intermediate fusion feature F m , t j represents the j-th eigenvalue in the image feature F i ;

[0084] S522. Then, use the SoftMax function to normalize the feature similarities S t and S i respectively to obtain the attention weights A t and A i , and the calculation formula is:

[0085] A t = SoftMax(S t )...(3)

[0086] A i = SoftMax(S i )...(4);

[0087] S523. Then, multiply the attention weights A t and A i with the intermediate fusion feature F m respectively to obtain the weighted fusion features F m t and F m i , and the calculation formula is:

[0088]

[0089] S524. Finally, use the Concat function to concatenate the weighted fusion features F m t and F m i to fuse and obtain the final fusion feature F f , and the calculation formula is:

[0090]

[0091] S60. The fusion feature F fIt is input into the downstream violation detector for detection, and the detection result is output.

[0092] The following uses the FCOS-R50 and YOLOF-R50 object detectors based on the ResNet-50 network to complete the downstream object detection task and conduct experimental comparisons. The experiment compares traditional single-modal object detection with the multi-modal graphic-text fusion object detection of the present invention, respectively for the above two object detectors. Model training and validation were carried out on the MSCOCO2017 dataset and the violation detection dataset constructed by the present invention. According to the experimental results in Table 1 and Table 2, it can be proved that the present invention improves the performance of object detection and market violation detection to a certain extent.

[0093] Table 1 Comparison of experimental result data of MSCOCO2017

[0094] Model mAP <![CDATA[AP 50 > <![CDATA[AP 75 > V1 38.7 58.0 39.5 V2 39.3 58.8 40.2 V3 39.5 58.6 40.2 V4 40.3 58.1 41.0

[0095] Table 2 Comparison of experimental result data of the violation detection dataset

[0096] Model mAP <![CDATA[AP 50 > <![CDATA[AP 75 > V1 85.2 89.2 86.6 V2 89.7 93.1 90.0 V3 88.4 91.0 86.9 V4 90.5 92.8 91.4

[0097] In the above table, V1 represents traditional single-modal object detection for the FCOS-R50 detector; V2 represents the present invention for the FCOS-R50 detector; V3 represents traditional single-modal for the YOLOF-R50 detector; V4 represents the present invention for the YOLOF-R50 detector; mAP represents the average precision of all category detections; AP 50 represents the average precision when the intersection over union (IOU) is 0.5, and AP 75 represents the average precision when the IOU is 0.75, where IOU is a concept in object detection, obtained by dividing the intersection of the detection box and the ground truth box by their union.

[0098] Through the above process, the present invention realizes the comprehensive and accurate detection of violations in market operations. Compared with the traditional single-modal detection method, the present invention improves the accuracy and reliability of market violation detection.

[0099] The above embodiments are only the preferred embodiments of the present invention, and do not limit the protection scope of the present invention. Any changes made by using the design principle of the present invention and non-creative labor on this basis shall fall within the protection scope of the present invention.

Claims

1. A market violation detection method based on multimodal image and text fusion, characterized in that: The following steps are involved: S10. Collect normal data and violation information in text and image formats in market operations; S20, pre-processing the collected normal data and violation information to construct a market business violation detection data set; S30. Use the ALBERT model to extract text features F from the market business violation detection dataset t , using the ResNet-50 network model to extract image features F from the market business violation detection dataset i ; S40, through the feature enhancer of the hybrid attention mechanism, the extracted text feature F t and image features F i Perform feature intra-modal enhancement and feature cross-modal enhancement alternately to obtain enhanced text feature F t ' and enhanced image features F i '; S50, the text feature F obtained in S30 t and image features F i Enhanced text feature F obtained with S40 t ' and enhanced image features F i 'Common input feature fusion, through the intermediate fusion and weighted fusion method to achieve F t 、F i 、F t ' and F i ' feature fusion to obtain the final fusion feature F f ; S60, fusion feature F f Input the downstream violation detector for detection and output the detection results.

2. The method for detecting market violations based on multimodal image-text fusion according to claim 1 is characterized in that: The process of preprocessing the collected normal data and violation information to construct a market business violation detection data set in step S20 is as follows: S21, sorting the image format data collected in the market business and numbering the image ID; S22, sorting the text-formatted data collected from the market business and storing it in a dictionary-formatted data structure in a key-value pair format, where the key corresponds to the image ID and the value corresponds to the text description; S23, dividing the data set into a training set, a validation set, and a test set according to the image ID; S24. Construct an annotation file to classify and label the image format data. The annotation file includes the image id, category id, annotation id, segmentation mask and target box corresponding to each image.

3. The method for detecting market violations based on multimodal image-text fusion according to claim 1 is characterized in that: In step S30, the ALBERT model is used to extract text features F from the market business violation detection dataset. t , using the ResNet-50 network model to extract image features F from the market business violation detection dataset i The process is: S31. Select the pre-trained ALBERT model. After inputting the text description corresponding to the specific id into the ALBERT model, the model outputs the embedded representation of the text, which captures the semantic information of the text. S32, load the ResNet-50 network model pre-trained on the large dataset ImageNet, input the image corresponding to the specific ID into the ResNet-50 network model, and the model extracts features layer by layer until the last convolution layer or global average pooling layer, and selects the output of the specific layer as the image feature; S33. Adjust the feature dimension by pooling text features and image features to extract key features.

4. The method for detecting market violations based on multimodal image-text fusion according to claim 1 is characterized in that: In step S40, the extracted text feature F is enhanced by a feature enhancer of a hybrid attention mechanism. t and image features F i The process of performing feature intra-modal enhancement and feature cross-modal alternating enhancement is as follows: S41. Text feature F t and image features F i The self-attention mechanism is used to achieve feature modality enhancement. The text feature adopts the multi-head self-attention mechanism of the Transformer architecture, and the image feature adopts the self-attention mechanism of the Non-local block. S42, the features enhanced within the feature modality are enhanced through an alternating co-attention mechanism to achieve alternating enhancement of features across modalities. The cross-modal attention mechanism of the Transformer architecture is adopted. The text features and image features enhanced within the modality are weighted, correlated, and weighted summed through a cross-modal Transformer encoder to obtain enhanced text features F t ' and enhanced image features F i '.

5. The method for detecting market violations based on multimodal image-text fusion according to claim 4 is characterized in that: The process of implementing the characteristic mode internal enhancement in step S41 is as follows: S411, the original text feature F t The multi-head self-attention mechanism of the Transformer architecture embeds the text features of the input sequence and then sends (Q, K, V) to the self-attention module of the encoder to calculate a new encoding representation. The multi-head self-attention mechanism divides (Q, K, V) into N parts, each of which is processed independently by the self-attention module, and finally the processing results of each part are fused, which can capture the multiple relationships and subtle differences between the input features. S412, the original image feature F i The self-attention mechanism of the Non-local block is adopted to enhance the features of useful and important information in the image. Through the non-local operation operator, only fewer stacking layers are needed to capture the long-range dependencies between the spatial positions of the image. Under the premise of ensuring the consistency of the input scale and the output scale, the calculation results can be more easily embedded into the current network structure.

6. The method for detecting market violations based on multimodal image-text fusion according to claim 4 is characterized in that: The process of implementing feature cross-modal alternating enhancement in step S42 is as follows: S421. The cross-attention mechanism can capture the internal feature structure and feature correlation between modalities. By inputting the text features and image features enhanced within the feature modality into the alternating co-attention mechanism, a cross-modal Transformer encoder is used to calculate the correlation between each region in the text features and image features, and the attention weight is used to measure this correlation. S422, adopt (Q i ,K t ,V t ) and (Q t ,K i ,V i ), takes text features and image features as Query queries, and then takes image features and text features as Key elements to match the queries. The attention weights are calculated through the cross-modal attention mechanism, and then the weighted sum operation is performed with the Value vector obtained by matching the query to obtain the enhanced representation features.

7. A method for detecting market violations based on multimodal image-text fusion according to any one of claims 1 to 6, characterized in that: In step S50, the text feature F obtained in S30 is t and image features F i Enhanced text feature F obtained with S40 t ' and enhanced image features F i 'Common input feature fusion, through the intermediate fusion and weighted fusion method to achieve F t 、F i 、F t ' and F i The process of feature fusion is: S51, through the intermediate fusion idea, the enhanced text feature F obtained in S40 t ' and enhanced image features F i 'Input into multiple fully connected layers, encode the correlation between modalities, and fuse to obtain the intermediate fusion feature F m ; S52, the text feature F obtained in S30 t and image features F i The intermediate fusion feature F obtained with S51 m Perform weighted fusion by calculating the text feature F t and image features F i and the intermediate fusion feature F m The similarity of is used to obtain the attention weight, and then the weighted feature is obtained through weighted fusion, and finally the weighted feature is fused through the concatenation function to obtain the final fusion feature F f .

8. The method for detecting market violations based on multimodal image-text fusion according to claim 7 is characterized in that: The process of weighted fusion in step S52 is as follows: S521, the text feature F t , image features F i and intermediate fusion feature F m For weighted fusion, first calculate the intermediate fusion features F by the cosine similarity formula m With the text feature F t The feature similarity S between t and intermediate fusion feature F m With image feature F i The feature similarity S between i , the calculation formula is: Among them, m i Represents the intermediate fusion feature F m The i-th eigenvalue in t i Represents the text feature F t The i-th eigenvalue in m j Represents the intermediate fusion feature F m The jth eigenvalue in t j Represents the image feature F i The j-th eigenvalue in ; S522, then use the SoftMax function to calculate the feature similarity S t and S i Normalize the features to get the attention weight A t and A i , the calculation formula is: A t =SoftMax(S t )……(3) A i =SoftMax(S i )……(4); S523, then the attention weight A t and A i Respectively with the intermediate fusion feature F m Multiply to get the weighted fusion feature F m t and F m i , the calculation formula is: S524, finally use the Concat function to combine the weighted fusion features F m t and F m i Splice and fuse to get the final fusion feature F f , the calculation formula is:

Images