A multi-modal information redundancy inference method in barrier-free crowd wisdom collaborative labeling

Abstract

The application discloses a kind of multi-modal information redundancy inference methods in barrier-free crowd wisdom collaborative labeling;The method comprises: data cleaning and matching to image, text and geographic location information in barrier-free labeling;Differential feature extraction is carried out to image, text and geographic location information to maximize the retention of key information in various modalities;Image features and text features are supplemented by geographic features to facilitate better alignment and fusion of image features and text features;Based on the twin network, a similarity inference method for paired labeling is proposed, so that the model can complete high-quality inference of redundant labeling in a short time.The application can greatly improve the accuracy and efficiency of redundant inference of barrier-free labeling, solve the redundant inference problem of labeling tasks dominated by geographic location information in general spatial crowdsourcing, and has broad market prospects and commercial value.

Description

Technical Field

[0001] This invention belongs to the field of computer-supported collaborative work technology, specifically, it relates to a method for inferring multimodal information redundancy in barrier-free collaborative annotation. Background Technology

[0002] Crowdsourcing is a model that distributes tasks to a large number of people through online platforms. In recent years, this model has not only received widespread attention and application in the business field, but has also been reflected in many aspects such as scientific research and social services. Examples include Meituan Waimai (food delivery platform), Didi Chuxing (ride-hailing service), dataset labeling and image classification in the field of artificial intelligence, and Wikipedia and accessibility facility labeling in the field of collaborative intelligence.

[0003] The main research issues in crowdsourcing tasks currently fall into four categories: First, quality control: Since crowdsourcing workers come from various fields and have varying skill levels, ensuring the quality of crowdsourcing tasks is crucial. Second, incentive mechanisms: Crowdsourcing tasks typically use monetary rewards, material incentives, or certificates to incentivize workers to complete assigned tasks efficiently and effectively; therefore, setting appropriate incentive mechanisms and reward distribution rules is also critical. Third, task allocation: Crowdsourcing tasks are allocated through online platforms; ensuring that a limited number of people can complete the assigned tasks more effectively and comprehensively is a major research issue for crowdsourcing platforms. Fourth, privacy and security: Privacy protection is an emerging research issue in crowdsourcing, especially with the development of mobile internet and spatial crowdsourcing tasks, making participants more concerned about their geographical location and other private information.

[0004] Spatial crowdsourcing is a significant and emerging branch of crowdsourcing. Its development is closely linked to the rise of mobile internet; in fact, it can be said that the development of mobile internet has driven the growth of spatial crowdsourcing tasks. The main differences between spatial crowdsourcing and traditional crowdsourcing lie in several aspects: geographical dependence (spatial crowdsourcing tasks typically require completion at a specific geographical location, while traditional crowdsourcing tasks are usually geographically independent); dynamism (spatial crowdsourcing often involves dynamic changes in time and location, increasing the complexity of task allocation and execution); real-time nature (due to geographical limitations, spatial crowdsourcing tasks usually need to be completed within a short period); participant mobility (spatial crowdsourcing typically requires participants to move to specific locations, involving more complex route planning and time management); and security and privacy (due to the involvement of geographical location information, spatial crowdsourcing places higher demands on participant privacy protection). However, existing research on quality control in spatial crowdsourcing lacks consideration of redundancy caused by subjective factors (malicious labeling, misunderstandings, etc.) and objective factors (network problems, inaccurate geolocation positioning, etc.), which can lead to quality issues.

[0005] Multimodal models take two or more modalities (e.g., images, text, video, audio, geolocation, etc.) as input, then use different feature extraction techniques to extract feature vectors, and finally fuse these features using a specific multimodal fusion model built for a specific task. Currently, some classic multimodal pre-trained models include ViLBERT, VisualBERT, Unicoder-VL, LXMERT, VL-BERT, UNITER, ImageBERT, Pixel-BERT, Oscar, and ViLT. Their main downstream tasks include visual question answering, image-text matching, image-text generation, multimodal sentiment analysis, and multimodal search. However, research on downstream tasks involving redundant inference for multimodal information fusion is currently lacking. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, the present invention aims to provide a method for inferring redundancy of multimodal information in barrier-free collaborative annotation. This invention can significantly improve the accuracy and efficiency of redundancy inference in barrier-free annotation, solve the problem of redundancy inference for annotation tasks dominated by geographic location information in general spatial crowdsourcing, and has broad market prospects and commercial value.

[0007] The technical solution of the present invention is described in detail below.

[0008] This invention provides a method for inferring multimodal information redundancy in barrier-free collaborative annotation, comprising the following steps:

[0009] (1) The image, text and geographic location information data generated in the collective intelligent collaborative annotation are cleaned by preliminary screening and formatting, and then the annotation information is randomly paired to complete the dataset construction;

[0010] (2) Differential feature extraction is performed on the image, text and geographic location information in each pair of paired annotations in the dataset to maximize the retention of key information in various modalities while obtaining the feature vector of the corresponding modality;

[0011] (3) Using geographic location information as supplementary information for image information and text information respectively, the feature vectors of the extracted image, text and geographic location modalities are fused to obtain the image fusion feature vector and text fusion feature vector of a single label in the paired label;

[0012] (4) Using the idea of ​​Siamese networks, the image fusion feature vector and text fusion feature vector of paired annotations are added, subtracted and multiplied respectively to learn the feature relationship between paired annotations. Then, the information obtained by adding, subtracting and multiplying different modalities are spliced ​​together. Based on the spliced ​​fusion vector, the similarity judgment of paired annotations is performed to realize the multimodal information redundancy inference in barrier-free collaborative annotation.

[0013] In this invention, in step (1), the pairing operation randomly encapsulates the labeled text information, image information, and geographic location information using an array, wherein the geographic location information is obtained by calculating the Euclidean distance.

[0014] In this invention, in step (2), text information is extracted using the BERT model, image information is extracted using the ResNet model, and geographic location information is extracted by calculating the position between paired labels using the Euclidean distance formula.

[0015] In this invention, in step (3), the image feature vector and text feature vector are first fused with the geographic information feature vector through a multi-head self-attention mechanism, and the geographic location information is used to supplement the image information and text information to obtain the corresponding image-pos vector and text-pos vector. Then, the obtained image-pos vector and text-pos vector are used for feature representation and feature fusion through a dual-head cross-fusion attention mechanism, so that the image information can carry a part of the text information and the text information can carry a part of the image information, thus obtaining the image-pos-text vector and the text-pos-image vector.

[0016] In this invention, in step (3), residual connections are introduced into the feature fusion processing network.

[0017] In this invention, in step (4), the probability distribution is calculated using the SoftMax function to determine whether the annotation is redundant.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention can complete the high-quality inference of redundant labels in the labeling task dominated by geographic location information in a short time, which greatly improves the accuracy and efficiency of redundant inference of accessibility labeling. Attached Figure Description

[0019] Figure 1 This is a flowchart of the deep learning-based multimodal training model of the present invention.

[0020] Figure 2 This is a flowchart of the feature vector extraction process for the multimodal training model of the present invention.

[0021] Figure 3 This is a diagram of the feature fusion module architecture of the multimodal training model of the present invention.

[0022] Figure 4 This is a structural diagram of the similarity inference method based on the multimodal model of Siamese networks in this invention. Detailed Implementation

[0023] This invention provides a method for inferring redundant multimodal information in accessible crowdsourced annotation. The method includes: data cleaning and matching of image, text, and geographic location information in accessible annotation; differential feature extraction of image, text, and geographic location information to maximize the retention of key information in various modalities; designing a multimodal information fusion model RCRBE based on LXMERT (where R represents residual, C represents cross, R represents ResNet-101 model, B represents BERT model, and E represents Euclidean distance; referred to as an accessible information multimodal deep learning model based on a dual-head cross-fusion mechanism with residual connections), allowing geographic features to supplement image and text features, promoting better alignment and fusion of images and text; and proposing a similarity inference method for paired annotations based on Siamese networks, enabling the model to complete high-quality inference of redundant annotations in a short time. This invention can significantly improve the accuracy and efficiency of redundant inference in accessible annotation, solving the problem of redundant inference in general spatial crowdsourcing tasks dominated by geographic location information, and has broad market prospects and commercial value.

[0024] Figure 1 This is a framework diagram of the present invention.

[0025] The multimodal information redundancy inference architecture of this invention mainly includes the following modules: a data collection and dataset creation module, a feature vector extraction module, a feature fusion module, and a similarity inference module. First, a large dataset containing images, text, and geographic location information needs to be collected through a spatial crowdsourcing platform. Then, this dataset undergoes machine and manual filtering to obtain basically usable accessibility annotation information. Next, the annotation information is randomly paired in pairs. Text and images only need to be placed in one array, while geographic location information needs to be calculated using Euclidean distance. (This corresponds to...) Figure 1 The dataset collection and processing section is as follows. Then, feature vectors are extracted from each pair of data to obtain the feature vectors of the corresponding modality. After obtaining individual feature vectors, a fusion module is used to fuse the feature vectors to obtain the correlation between image, text, and geographic location feature vectors. Finally, the fused vectors are used by a similarity inference module to obtain the corresponding redundancy results.

[0026] Figure 2 This is a flowchart of the feature vector extraction process of the present invention.

[0027] from Figure 2 As seen in the diagram, image information is processed using a ResNet model for feature vector extraction. Specifically, the ResNet101 network is used. Feature vector extraction is the most time-consuming of the three modalities. The ResNet101 network can efficiently and effectively extract key information from images, improving the performance and efficiency of model training. Text information is extracted using a BERT model. In the field of NLP machine learning, the BERT model is widely recognized as the optimal model for processing text information. Choosing the BERT model here effectively captures the context and key information of the text, thus improving the accuracy of the model results. Geographical location information is extracted using Euclidean distance. Euclidean distance effectively reflects the correlation between annotations and can complement the text and image feature vectors.

[0028] Figure 3 This is the feature fusion module architecture of the present invention.

[0029] The feature fusion module architecture primarily utilizes multi-head self-attention and dual-head cross-attention mechanisms. Multi-head self-attention is beneficial for capturing the fused geographic location information and dependencies between images and text; it supplements image and text information with geographic location information, enabling information alignment. Dual-head cross-attention, on the other hand, mainly affects the fusion of image and text feature vectors, allowing them to integrate each other's information and learn information from other dimensions, thus improving the model's expressive power. Figure 3After the fusion architecture is completed, it can effectively map and supplement the text and image information with geographic information. At the same time, it can enable images and text to learn the feature representation of another modality from each other, so that the feature vectors can be learned and fused to the greatest extent, thereby enabling the similarity inference module to have better accuracy.

[0030] The feature fusion module architecture incorporates residual connections, which help optimize information flow during training and prevent information loss in deep networks. Residual connections allow the model to directly pass features from previous layers, helping to preserve the integrity of existing features while learning new ones. Before fusing geographic information with image and text features, each feature vector is residually connected to the corresponding features from previous layers before being input into the attention mechanism. This design allows the model to retain the original feature vectors of images and text while fusing new contextual information. The use of residual connections enhances the model's ability to remember original input features, improves the effectiveness of feature fusion, and contributes to the accuracy and robustness of the final inference. This feature fusion strategy based on residual connections strengthens the association between image and text features and geographic information while effectively avoiding information loss caused by information compression during the fusion process.

[0031] Figure 4 The similarity inference module of this invention is based on the idea of ​​Siamese networks. It pairs the labels, then uses model structures with the same structure and weights, and finally splices and fuses the resulting fusion models to complete the redundant inference results through similarity judgment.

[0032] The technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0033] Example 1

[0034] A method for inferring multimodal information redundancy in barrier-free collaborative annotation, the specific steps of which are as follows:

[0035] Step 1: Collect and build training and testing datasets

[0036] First, multimodal accessibility information data generated from the collaborative annotation platform was collected, and invalid annotations were removed. Then, the initially screened accessibility information underwent basic data cleaning using Python tools. Next, the text and image modal information was further processed: 20% of the text information was randomly masked; geographic location information was accurate to the meter; and images were numbered, grayscale processed, and resized to 384*384. Finally, the processed data were randomly paired, and 80% of the total data was used as the training set for the model, while the remaining 20% ​​was used as the test set.

[0037] The pairing mentioned above refers to pairing two accessibility labels (including images, text, and geographic locations), where the geographic information is processed directly using Euclidean distance; the paired format is shown in the example below:

[0038] {text:[“This is an accessible elevator”,“The accessible elevator is located on Line 10”],image:[image1.jpg,image2.jpg],position:0.8932131}

[0039] Step 2: Multimodal Information Feature Extraction

[0040] Based on the paired data obtained in step 1, feature vectors are extracted from individual modalities using the target model. Text information is extracted using Chinese-Pert-Base to adapt to the Chinese attributes of the annotation information; image information is extracted using a ResNet-101 network; and geographic location information is calculated using the Euclidean distance formula to determine the positional features between paired annotations P and Q. It is important to note that the paired text and image information must be processed by the same feature vector encoder to ensure that the model parameters and weights are identical; otherwise, it will affect the subsequent feature vector fusion and concatenation process.

[0041]

[0042] Step 3: Multimodal Fusion Model Architecture Design

[0043] When constructing the multimodal fusion model, a multi-head attention mechanism and a two-head cross-attention mechanism were used. The multi-head self-attention mechanism is mainly to allow geographic location information to supplement the feature vectors of image and text information, avoiding excessive differences between the image and feature vectors after fusion. The two-head cross-attention mechanism is used to perform feature representation and feature fusion of the processed image feature vectors and text feature vectors, so that image information can carry some text information, and text information can carry some image information, thereby achieving better feature fusion and learning the semantic representation of another modality.

[0044]

[0045]

[0046] The specific processing steps are as follows:

[0047] 1. First, the obtained image vector and text vector are fused with the geographic information feature vector using a multi-head (8-head) self-attention mechanism to obtain the corresponding image-pos vector and text-pos vector.

[0048] image-pos=Attention(pos_f,image_f,image_f)

[0049] text-pos=Attention(pos_f,text_f,text_f)

[0050] MultiHeadOutput = Concat(h 1 ,h 2 ,…,h H W O

[0051] 2. Next, the obtained image-pos vector and text-pos vector are used for feature representation and feature fusion through a dual-head cross-fusion attention mechanism to obtain image-pos-text vector and text-pos-image vector.

[0052] image-pos-text = Attention v-to-t (image-pos,text_f,text_f)

[0053] text-pos-image = Attention t-to-v (text-pos,image_f,image_f)

[0054] Step 4: Redundancy inference based on Siamese networks

[0055] This method uses Siamese networks for similarity judgment to infer the similarity of redundant annotations. To facilitate comparison of results using the same network structure, pairing of annotations is performed from the dataset construction stage. Then, the paired multimodal information is passed through the same encoder and fusion model to ensure consistency of weights and parameters during the training process of the paired annotations. Finally, the two fusion vectors obtained are concatnized, and the SoftMax function is used to infer whether the paired annotations are redundant.

[0056] concat_f=[emb1,emb2,|emb1-emb2|]

[0057] prediction = SoftMax(output)

[0058] The SoftMax function converts the feature vector similarity scores calculated from paired annotations into a probability distribution. Each element in the vector is assigned a probability value between 0 and 1, and the sum of all probability values ​​is 1. In this invention, redundancy judgment can be understood as a binary classification operation. The array is [x, y], where x represents redundancy and y represents non-redundancy. The softmax function assigns a probability value to both the x and y vectors, for example, [0.4, 0.6], and selects the larger value between the two as the inference result.

[0059] Furthermore, this invention independently constructed a dataset based on accessibility information collected through crowdsourcing and conducted experiments. Experimental conditions:

[0060] The experiment used a deep learning model deployed on a T4 server. During training, the batch size was set to 32, the learning rate was adjusted to 2e-5, AdamW was used as the gradient descent optimization algorithm, and the weight decay was set to 0.02. The cross-entropy loss function was chosen. Training lasted for 30 epochs, and an early stopping strategy was implemented to prevent overfitting. The ReLU activation function was selected.

[0061] Metrics: Accuracy, Loss, Recall, Precision, F1 Score;

[0062] Table 1

[0063]

[0064] The results are shown in Table 1. The accuracy of this invention is 0.990; the loss value is 0.350; the precision is 0.982; the recall is 1.000; and the F1 score is 0.991. The results show that under multimodal input conditions, this invention exhibits high accuracy and precision, especially when the model input simultaneously includes image, text, and location information. For example, the LXMERT and RCRBE models demonstrate accuracy and precision exceeding 95%, and a recall of 100%, showing the model's high sensitivity to correct annotation. Furthermore, the decreasing trend of the loss value confirms the effectiveness of the model training process and the rationality of the optimization algorithm. The experimental results fully verify the superior performance of the RCRBE model designed in this paper in the task of detecting redundancy in accessibility information, especially its efficiency and accuracy in handling complex multimodal data.

Claims

1. A method for inferring multimodal information redundancy in barrier-free collaborative annotation, characterized in that, Includes the following steps: (1) The image, text and geographic location information data generated in the collaborative annotation process are cleaned by preliminary screening and formatting, and then the annotation information is randomly paired to complete the dataset construction; (2) Differential feature extraction is performed on the image, text and geographic location information in each pair of paired annotations in the dataset to maximize the retention of key information in various modalities while obtaining the feature vector of the corresponding modality; (3) Using geographic location information as supplementary information for image information and text information respectively, the feature vectors of the extracted image, text and geographic location modalities are subjected to feature fusion processing to obtain the image fusion feature vector and text fusion feature vector of a single label in the paired label; (4) Utilizing the concept of Siamese networks, the image fusion feature vector and text fusion feature vector of paired annotations are added, subtracted, and multiplied respectively to learn the feature relationship between paired annotations. Then, the information obtained from the addition, subtraction, and multiplication of different modalities are concatenated. Based on the concatenated fusion vector, the similarity judgment of paired annotations is performed to realize the inference of multimodal information redundancy in barrier-free collaborative annotation; where: In step (2), text information is extracted using the BERT model, image information is extracted using the ResNet model, and geographic location information is extracted using the Euclidean distance formula to calculate the position between paired labels. In step (3), the image feature vector and text feature vector are first fused with the geographic location information feature vector through a multi-head self-attention mechanism. The geographic location information is used to supplement the image information and text information to obtain the corresponding image-pos vector and text-pos vector. Then, the obtained image-pos vector and text-pos vector are used for feature representation and feature fusion through a dual-head cross-fusion attention mechanism, so that the image information can carry some text information and the text information can carry some image information, resulting in the image-pos-text vector and text-pos-image vector. In step (3), residual connections are introduced into the feature fusion processing network.

2. The method for inferring multimodal information redundancy in barrier-free collaborative annotation according to claim 1, characterized in that: In step (1), the pairing operation randomly encapsulates the labeled text information, image information, and geographic location information using an array, where the geographic location information is obtained by calculating the Euclidean distance.

3. The method for inferring multimodal information redundancy in barrier-free collaborative annotation according to claim 1, characterized in that: In step (4), the probability distribution is calculated using the SoftMax function to determine whether the annotation is redundant.

Images