Mobile crowd sensing image truth value discovery method based on open set target detection

Abstract

In recent years, an image-based perception normal form appears in the field of mobile crowd sensing, and an efficient and low-cost solution is provided. However, the current work cannot effectively detect the authenticity of the image data, which threatens the security of the crowd sensing platform. And data between malicious users and low-quality perception users are different, but the characteristic of perception user data cannot be effectively utilized in previous data detection. In order to solve the problems, the invention provides a method based on open set target detection, and aims to detect perceived image information and provide an independent distinguishing method for malicious data and low-quality data so as to reduce platform load and improve platform availability. The method comprises the following steps: firstly, extracting target image data by using an open set target detection method to obtain a feature vector of an area where a sensing target in a sensing image is located; secondly, aiming at the arbitrariness of malicious data distribution, providing a method of using a threshold value to segment a vector space of a sensing target and other vector spaces; and finally, aimed at the high similarity between low-quality data vector distribution and a perception target, feature vectors of different types of similar targets are utilized to assist truth value discovery. Finally, efficient recognition of the sensing image is realized, and the data security and reliability of the platform are improved, so that the overall performance and user experience of the crowd sensing platform are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of mobile crowd sensing, and in particular relates to a method for discovering true values ​​of mobile crowd sensing images based on open set target detection. Background Art

[0002] Mobile Crowd Sensing (MCS) systems rely on data actively contributed by participants' mobile device sensors. However, due to the openness of MCS systems, malicious users often upload low-quality and erroneous data to defraud rewards, resulting in increasingly prominent data quality issues. Therefore, truth discovery has become a hot topic in recent years. In order to maintain the integrity and reliability of collected information, an effective detection system is urgently needed to mitigate the impact of malicious sensor data. At present, many methods have been proposed to solve the truth discovery problem. Some of the simpler techniques include the average method, the median method, and the weighted average method. In addition, there are studies that use neural networks to distinguish benign and malicious data from the SherLock dataset containing thousands of records and 238 sensor features. Current MCS data detection methods are rich and varied, mainly focusing on the processing of structured data. Structured data, also known as quantitative data, is information that can be represented by a unified structure, such as numbers and symbols. This type of data is usually stored in a relational database, and the internal structure is determined by metadata or fixed fields. Although the clear relationship of structured data makes its application very convenient, it is relatively low in terms of commercial mining value. In recent years, a series of perception methods based on unstructured data such as images have emerged to address the pain points of small amount and low value of structured data. This not only enhances the perception ability of the MCS system, but also further increases the urgency of MCS security issues. As unstructured data, images have no fixed format, and processing and detection are more complicated, and malicious data is more likely to be confused. When users cannot obtain sufficient rewards from the platform, the user's willingness to perceive will decrease, making it impossible for the platform to obtain perception data. Therefore, it is necessary to propose an image-based MCS false data detection method. Using an object detection model is an effective response strategy. Object detection is a computer vision method that usually consists of a backbone network, a classification head, and a position regression. The backbone network extracts the features of the MCS perception image, and the classification head and position regression output the category information of a certain position. However, the categories output by traditional object detection are discrete one-hot encodings, which limits it to only detect a limited number of categories and cannot adapt to the diverse tasks required by the MCS system as an open platform. In the field of classification, the CLIP model is a multimodal model that maps images and texts to the same space through comparative training of text and images, and then determines the semantic information of the image. CLIP is able to classify any image due to extensive image-text pair comparison training, but its embedding is an abstraction of the entire image. In the MCS system, the standards of crowdsourced images vary, and the perceived target may appear anywhere in the image. If the entire image is extracted as an embedding, the information obtained will be quite different from the target.

[0003] To solve the above problems, the open set target detection model is a new method that can better extract feature vectors in the region.Figure 1 As shown in the figure, open set target detection is between target detection and CLIP, and the output target is a continuous high-dimensional vector space. Compared with the one-hot method of traditional target detection, open set detection can extract regional continuous information. For example, the open set detection model encodes specific areas in the image, thereby more accurately reflecting the semantic information of the target in the image. In addition, there are two reasons for some non-genuine data: one is malicious data, that is, users cheat rewards by uploading random data; the other is low-quality data, which is usually caused by user operational errors in the perception scene or unfamiliarity with the perception target. Past MCS truth discovery methods have failed to distinguish different types of non-genuine data causes, and this defect needs to be improved urgently. In response to these problems, proposing an image-based open set target detection method will be the key to enhancing the effectiveness of the MCS system. The current challenges in the field of data detection are as follows:

[0004] (1) There is currently no detection method for image data in view of the MCS image perception paradigm that has emerged in recent years, which seriously affects the security and reliability of MCS data.

[0005] (2) Current detection methods do not distinguish between malicious data and low-quality data. They ignore the fact that malicious data is distributed in arbitrary space, while low-quality data is distributed in locations close to the perceived target. Summary of the invention

[0006] In view of the above-mentioned problems, the present invention proposes a method for discovering the true value of mobile crowd perception images based on open set target detection to solve the need for discovering the true value of crowd perception images. This method is based on open set target detection to realize the detection of arbitrary targets in crowd perception images. By distinguishing between two types of non-true data, a higher accuracy rate can be effectively obtained. In malicious data monitoring, the distance between high-dimensional vectors is used to determine whether it is real data. In low-quality data monitoring, multiple similar objects are used to assist in the true determination of perceived targets. The overall framework of MCS image detection is as follows: Figure 2 As shown, it mainly includes the following three parts:

[0007] (1) An open set detection model for detecting image data in MCS system is proposed. The output of the detection model is a region information and a feature vector representing a continuous space, which can capture the semantic information of the region.

[0008] (2) A detection method for malicious images in the MCS system is proposed. This method can effectively distinguish the perceived target from other arbitrary images and prevent perceived users from uploading meaningless images to the MCS system to defraud rewards.

[0009] (3) A detection method for low-quality image data in the MCS system is proposed. This method can effectively detect the poor perceived image quality caused by user errors, failure to identify perceived targets, etc.

[0010] Compared with the prior art, the beneficial effects of the present invention are: (1) On the basis of traditional structured data detection, the function of detecting unstructured data such as images is added. Detecting legitimate target image data and filtering out other image data can greatly alleviate the pressure on the MCS platform. (2) Proposed detection of malicious data. By analyzing the distribution of malicious image data, a metric specifically used to measure real data and malicious data is designed. Improve the ability to detect malicious data. (3) The distribution characteristics of detection of low-quality data, that is, users will collect and upload similar targets as perception targets, or other related targets that may appear in the perception environment, use a single distance measurement is imperfect. Therefore, different types of similar objects are used to assist in true value discovery, thereby improving detection capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a comparison chart between CLIP and open set target detection;

[0012] Figure 2 It is the overall structure diagram of the image truth discovery model;

[0013] Figure 3 It is a UMAP visualization of image data detection by the open set detection method;

[0014] Figure 4 This is a UMAP visualization of the CLIP method’s detection of contrasting image data. DETAILED DESCRIPTION

[0015] The input of the MCS-aware image detection method proposed in the present invention consists of an image-text pair, and the output is the position and embedding of the noun in the image corresponding to the noun described in the text. Figure 2As shown, the model of the present invention is a dual encoder and single decoder structure. It includes a Bert text encoder and a Swin transformer image encoder. Bert is responsible for converting the text description to be detected into a text vector, and the Swin transformer is responsible for extracting the feature map composed of the image vector from the picture. After the variable self-attention transformation, the cross-attention of the text vector to the image and the cross-attention of the image to the text are modally fused. Finally, according to the similarity between the image vector and the text vector, 900 similar vectors are selected for decoding. The decoding result is the regional feature vector embedding and the selection box position information. Finally, malicious data detection or low-quality data detection is adopted according to the application needs. The following will be a specific description of each component module.

[0016] Image encoder: The image encoder structure proposed in this invention is based on the hierarchical transformer and has powerful feature extraction capabilities. The following is the formal definition of the image encoder:

[0017] Step 1: Input and embedding, the input image is divided into H×W non-overlapping patches of size P×P, and each patch P is flattened. A linear layer is used to transform it into a vector of dimension d: PatchEmbedding(I) d =Linear(Flatten(P))

[0018] Step 2: The Hierarchical Feature Extraction model uses multiple stages, each with different patch sizes and resolutions, with each layer’s resolution decreasing gradually. At the end of each stage, adjacent features are merged using a Patch Merging layer, turning H×W vectors of dimension d into H / 2×W / 2 vectors of dimension 2d, while increasing the feature dimension. A linear layer is then used to reduce the dimension to the original consistency, and finally X∈R H×W×D Transformed into X∈R H / 2×W / 2×D .

[0019] Step 3: SWMSA (Shifted Window Multi-Scale Attention) aims to improve efficiency and performance in image processing tasks. Here are some key features of SWMSA: SWMSA uses a local window approach to calculate self-attention, which can significantly reduce computational complexity and can more effectively process large-size images compared to global self-attention. At each stage, the position of the window is shifted, which ensures that the model can capture richer contextual information at different levels. This shift mechanism allows information between different windows to interact, improving the expressiveness of the model. SWMSA can extract features at multiple scales, enhancing the model's ability to recognize objects of different sizes. This is especially important for processing complex scenes and images with rich details. Specifically, the window position moves relatively in the calculation of each Block. For example, if B represents the relative position bias matrix, Q, K, and V are the query vector, key vector, and value vector in the current window, respectively. d is its dimension, which is defined as follows:

[0020] Text encoder: The text encoder proposed in this paper is a pre-trained language model based on the Transformer architecture, mainly used for natural language processing tasks. Its formal definition can be summarized into the following key components:

[0021] Step 1: For the input sequence X = [x1, x2, ..., x n ], the input of the text encoder consists of the following three parts. Token IDs: each token x i is mapped to a unique ID vector t i , we get the input vector T = [t1, t2, ..., t n ]. Position encoding: Assign a position ID to each token and get the position vector P = [p1, p2, ..., p n ] where p i is x i The position index of the first layer. 1 It is expressed as: H (1) =T+P=[t1+p1,t2+p2,...,t n +p n ]

[0022] Step 2: The encoder is composed of L layers of Transformer encoders stacked together. Q (query), K (key), V (value) are H (L -1) Through linear transformation, d kis the dimension of the key. The output H of each layer (L) Defined as:

[0023] Step 3: Output representation. After passing through L layers of Transformer, the final output is H L =[h1,h2,...,h n ].

[0024] Feature Fusion: Given an image-text pair, we use the image encoder to extract multi-scale image features and the text encoder to extract text features. After extracting the original image and text features, we feed them into the feature enhancer for cross-modal feature fusion. The feature enhancer includes multiple feature enhancement layers, namely deformable attention and cross attention between image and text features.

[0025] Deformable Attention is a technique used to improve the flexibility and adaptability of the attention mechanism, especially when dealing with inputs with complex shapes and diverse structures. The following is its detailed formal definition: For an input feature map, where X∈R C×H×W , where C is the number of channels, H and W are the height and width of the feature map. Deformable attention introduces the learned offset ΔP to dynamically adjust the attention position, P′=P+ΔP, and calculates the attention weight at the new position P. The attention weight A is normalized by the SoftMax function.

[0026] Step 1: Calculate the offset attention weight:

[0027] The attention weight is calculated at the new position P'. If Q and K are the feature representations of the query and key respectively, the calculation is performed at the offset position P'. The attention weight A can be normalized by the SoftMax function: A=SoftMax(QK T )

[0028] Step 2: V k is the value feature, A k is the corresponding attention weight. The final output feature Y is obtained by weighted summation:

[0029] Step 3: An important feature of deformable attention is the ability to perform deformable convolution, which allows focusing on a specific target area. At this point, W(k) is the convolution kernel weight, Δh(k) and Δw(k) are the learned offsets, and the convolution operation becomes:

[0030] Cross-attention between image and text features: Cross-attention is used to deal with the alignment problem of text feature vector and image feature vector in this invention. It enhances information expression by calculating the relationship between the two modalities. The following is the formal definition of cross-attention: Assume that we have two input modalities, image feature P∈R n×d and text features T∈R m ×d , where n and m are the number of image and text features respectively, and d is the dimension of the feature.

[0031] Step 1: Use dot product to calculate the similarity between image features and text features. Get the attention score matrix A∈R m×n , where Q and K are learnable transformation matrices.

[0032] Step 2: Normalize and calculate the weighted feature representation, normalize each row to obtain the normalized attention weight matrix P∈R m×n , use the attention weights to weight the image features and get the weighted image representation Z∈R for each text feature m×d .

[0033] Feature selection: This paper represents image features as The text features are represented as N I Indicates the number of image markers, kept below 256. N T represents the number of text tags, which is 10,000. d corresponds to the feature dimension, which is 256. The goal is to extract N q As the input of the decoder, N q The value of is set to 900. The Max function performs the maximum operation along the -1 dimension, and the symbol represents the matrix transpose, Indicates the operation of selecting the first Nq indexes. The extraction formula is as follows:

[0034] Cross-modal Decoder: We develop a cross-modal decoder to combine image and text modality features. Each cross-modal query is input into a self-attention layer, an image cross-attention layer for combining image features, a text cross-attention layer for combining text features, and an FFN layer in each cross-modal decoder layer. Compared with the standard decoder layer, each decoder layer has an additional text cross-attention layer because we need to inject text information in the query for better modality alignment.

[0035] Input vector space: Let the perceived image input space be x∈R H×W×C , where H represents the image height, W represents the image width, C represents the number of channels (C = 3 in RGB images), and each element x∈X is an image instance. That is: X={x|x∈R H×W×C}

[0036] Output vector space: Let the output space be Y∈R d , where d is the common dimension of the text vector and the image vector. The category y∈Y is represented by a vector center C y ∈R d The set of all categories constitutes the output space Y, C is the set of categories, c y is the center vector of each category y. d represents the dimension of the text vector. That is:

[0037] There are two reasons for the non-genuine data: one is malicious data for defrauding rewards, and the other is low-quality data generated by mistake. In the present invention, the two types of low-quality data are processed separately.

[0038] Detection of malicious data: Some users will post some completely meaningless data in order to defraud rewards, so it is necessary to develop a separate detection method for this part of data. Figure 3 As shown, completely meaningless image data may consist of any distribution. The detection form for malicious data is defined as follows:

[0039] Discriminant function: The discriminant is based on the input vector X∈R d With category center c y ∈Y cosine similarity. If the cosine similarity between the input vector and the center of a category is greater than or equal to a threshold τ, the input is considered to belong to that category. The output vector c x The conditions for being judged as category y are:

[0040] Detection of low-quality data: low-quality data caused by user errors and other reasons. Figure 4 As shown in the figure, due to the complex relationship between the detection target and the environment, similar detection targets may appear in the same environment. This section starts from the missing or misdetected perception targets in the image. The following formal definition is proposed. Suppose the goal of a perception task is to detect a certain type of specific object, denoted as the target category, and the task is completed by the user at the perception location. Due to operational errors (such as shaking, misalignment, and not recognizing the perceived target, etc.), the target category C is not effectively captured in the image I uploaded by the user. T , but the image still contains other objects in the environment, which are called non-target categories CNT , these objects may be random objects in the environment, with similar relevance to the target. To deal with this situation, the goal of the task is transformed into finding objects with the target category C from image I. T The object with the greatest similarity. Discriminant function: Unlike malicious data, the detection of low-quality data requires the classification of multiple targets. For example, for some similar animals, the use of threshold methods may cause a donkey to be identified as a horse. Multiple classification heads can effectively distinguish these similar classifications. The formal definition is as follows:

[0041] Through the above-mentioned mobile crowd sensing image truth value discovery method model based on open set target detection, the image data collected by the crowd sensing system can be effectively detected. And by distinguishing between the two types of non-true data, the detection accuracy is improved and the non-true data of the system is minimized.

[0042] Although the above describes the specific implementation mode of the present invention in conjunction with the drawings, it is not intended to limit the scope of protection of the present invention. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and should be included in the scope of protection of the present invention.

Claims

1. A method and system for detecting true value of mobile crowd-sensing images based on open set target detection, characterized in that: Including detection model structure, methods for malicious data detection and low-quality data detection.

2. The method for discovering true value of crowd-sensing images based on open set target detection according to claim 1, characterized in that: According to the text task description given by the user who publishes the mobile crowd sensing task, the method can determine whether the image contains the sensing target described in the text, and thus determine whether the data uploaded by the sensing user meets the description requirements.

3. The method for detecting malicious data by crowd intelligence based on open set target detection according to claim 1 is characterized in that: When detecting malicious image data, the mobile crowd-sensing image truth detection method based on open set target detection will map the image to a latent space that coexists with text data. In this space, the cosine distance described in this article is used to divide it into two parts, one part is benign data, and the rest is malicious data.

4. The method for detecting low-quality data based on crowd intelligence perception integrating user personality structure according to claim 1, characterized in that: When detecting low-quality image data, the mobile crowd-sensing image truth detection method based on open set target detection will map the description texts of other possible targets in the environment to a common image and text space to finely distinguish similar perceived targets.

Images