Efficient query method based on perceptual multimodal big data
Through the efficient query method of perceived multimodal big data, the feature extraction and transfer learning of image and text data are used to solve the shortcomings of data rationality judgment in water source management, and the efficient evaluation of image and text data is achieved.
Patent Information
- Application Number
- CN202011547371.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-24
AI Technical Summary
The prior art cannot effectively judge the rationality of image sets or text sets in water source management, especially in data processing in the same area of the detection data.
Using an efficient query method based on perceptual multimodal big data, by collecting image modal data sets and generating point cloud data, combining text modal data sets, using CountVectorizer and CAA algorithm, a training table for image feature-text features is established and the migration is carried out. Learn, generate a query model, and use the model to determine the characteristics of subsequent graphics or text to judge the qualification of the data.
It realizes efficient feature extraction and classification of image and text data, can infer and judge based on the detection data, and improves the evaluation efficiency of data rationality.
Smart Images

Figure CN112732956B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data query, and in particular to an efficient query method based on perceptual multimodal big data. Background Art
[0002] The so-called "modality" in English is modality, which means "sense" in layman's terms. Multimodality is the integration of multiple senses. The Turing OS robot operating system defines the interaction mode between robots and humans as "multimodal interaction", that is, human-computer interaction through multiple methods such as text, voice, vision, action, and environment, fully simulating the interaction between people. This interaction method is in line with the morphological characteristics of robot products and user expectations, breaking the traditional PC-style keyboard input and smartphone touch-based interaction mode.
[0003] In the field of water resource management, automated control is usually used, including video monitoring, depth drop detection, water quality detection, etc. Whether it is video monitoring or depth drop detection, an image set or a text set will be generated.
[0004] The existing method of processing these image sets or text sets is through human observation or setting a maximum value alarm. It is impossible to judge the rationality of the detection data of other same areas based on the existing data. Summary of the invention
[0005] In order to overcome the deficiencies in the prior art, the present invention proposes an efficient query method based on perceptual multimodal big data, which can infer subsequent data based on the detection data.
[0006] In order to achieve the above-mentioned object, an efficient query method based on perceptual multimodal big data of the present invention comprises the following steps: step 1: collecting an image modality data set, filtering the image and generating point cloud data;
[0007] Step 2: According to the image depth value of the point cloud data in step 1, obtain the feature values of all points under the point cloud data; Step 3: Asynchronously collect the text modality dataset and use CountVectorizer to extract features from the text;
[0008] Step 4: Use the CAA algorithm to establish a training table of image features and text features; Step 5: Randomly classify the data in the training table to generate a training set database and a test set database, and perform transfer learning on the data in the training set database and the test set database; Step 6: Based on transfer learning, establish a query model and upload it to the cloud. Subsequent query graphics or text input the query model, and the query model will identify the features of subsequent graphics or text, and query on the Internet based on the identified features.
[0009] Furthermore, in step 1, the variance method is used for filtering, and the formula is as follows:
[0010]
[0011] in, is the ratio of background points to image, is the average grayscale of the background points in the image, is the total average grayscale of the image; is the variance of the image.
[0012] Furthermore, in step 2, the threshold method is used to obtain the image depth value, and the formula is as follows:
[0013] p(x,y)=p(x,y) dis≤p(x,y)≤dis*2h,
[0014] Among them, p(x, y) is the image depth, dis is the height of the image, and h is the height of the filtered image.
[0015] In step 2, the image depth is collected to form point cloud data, and the point cloud data is processed using PCL to obtain the characteristic value of the image depth.
[0016] Furthermore, in step 3, the feature selection frequency formula of CountVectorizer is as follows:
[0017]
[0018] Among them, IDF is the feature selection probability, m is the number of point cloud data, and g is the number of eigenvalues.
[0019] Furthermore, the ratio of the training set database to the test set database is 1:2.
[0020] Beneficial effect: The feature values of the image modality dataset and the text modality dataset are retrieved, and then transfer learning is used to build and train a model for the feature values. The subsequent measured image set or text set is then input into the transfer learning model to determine whether the subsequent image set or text set meets the surface of the transfer learning model, so as to determine whether the data of the image set and text set are qualified. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The present invention will be further described and explained below in conjunction with the accompanying drawings.
[0022] Figure 1 It is a flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0023] The technical solution of the present invention will be more clearly and completely explained below through description of preferred embodiments of the present invention in combination with the accompanying drawings.
[0024] like Figure 1 As shown, the preferred embodiment of the present invention is an efficient query method based on perceptual multimodal big data, including step one: collecting image modality data sets, filtering images and generating point cloud data.
[0025] In the field of water source management, there are generally two types of modal data collected: image modality and text modality. Image modality includes videos and photos taken by the monitoring system. Text modality includes measured depth values, water quality values, etc.
[0026] Point cloud data refers to a set of vectors in a three-dimensional coordinate system. Scanning data is recorded in the form of points, each of which contains three-dimensional coordinates, and some may contain color information (RGB) or reflection intensity information (Intensity).
[0027] Step 2: According to the image depth value of the point cloud data in step 1, obtain the feature values of all points under the point cloud data.
[0028] For each data point in the point cloud, the least squares local plane p is fitted through its K local neighboring points, so that the sum of the distances from all the neighboring points of the data point to this plane is minimized.
[0029] Step 3: Asynchronously collect text modality datasets and use CountVectorizer to extract features from the text.
[0030] The CountVectorizer() function only considers the frequency of each word; then it forms a feature matrix, where each row represents the word frequency statistics of a training text. The idea is to first consider all the training texts, regardless of their order of appearance, and only regard each word that appears in the training text as a column of features to form a vocabulary.
[0031] Step 4: Use the CAA algorithm to create a training table of image features and text features.
[0032] Computer-aided numerical analysis (CAA) and computer-aided design (CAD) are analysis and design methods that use computers as the main tool. They are an emerging discipline based on computing technology, applied mathematics and simulation theory, and have become an important branch in the field of computer applications. The rapid development of science and technology has made the mathematical models established in scientific theory research, new product development and engineering design increasingly complex.
[0033] Step 5: Randomly classify the data in the training table to generate a training set database and a test set database, and perform transfer learning on the data in the training set database and the test set database.
[0034] Instance-based transfer learning studies how to select instances from the source domain that are useful for training in the target domain. For example, it can effectively assign weights to labeled data instances in the source domain so that the source domain instance distribution is close to the target domain instance distribution, thereby establishing a reliable learning model with high classification accuracy in the target domain.
[0035] Because the data distribution of the source domain and the target domain in transfer learning is inconsistent, all labeled data instances in the source domain are not necessarily useful for the target domain.
[0036] Step 6: Create a query model based on transfer learning and upload it to the cloud. Subsequent query graphics or text are input into the query model. The query model identifies the features of subsequent graphics or text, and queries are performed on the Internet based on the identified features.
[0037] After the results of transfer learning are uploaded, it is convenient to judge different data multiple times in multiple places.
[0038] The above specific implementations are only descriptions of the preferred implementations of the present invention, and do not limit the protection scope of the present invention. Without departing from the design concept and spirit of the present invention, various modifications, substitutions and improvements made by ordinary technicians in this field to the technical solution of the present invention based on the text description and drawings provided by the present invention should all fall within the protection scope of the present invention. The protection scope of the present invention is determined by the claims.
Claims
1. An efficient query method based on perceptual multimodal big data, characterized in that: The following steps are included: Step 1: Collect image modality data sets, filter the images and generate point cloud data, and use the variance method for filtering. The formula is as follows: in, is the ratio of background points to image, is the average grayscale of the background points in the image, is the total average grayscale of the image; is the variance of the image; Step 2: According to the image depth value of the point cloud data in step 1, obtain the feature values of all points under the point cloud data, and use the threshold method to obtain the image depth value. The formula is as follows: p(x,y)=p(x,y) dis≤p(x,y)≤dis*2h, Among them, p(x, y) is the image depth, dis is the image height, and h is the height of the filtered image; Collect image depth to form point cloud data, use PCL to process the point cloud data and obtain the characteristic value of image depth; Step 3: Asynchronously collect text modality datasets and use CountVectorizer to extract features from the text; Step 4: Use the CAA algorithm to create a training table of image features and text features; Step 5: Randomly classify the data in the training table to generate a training set database and a test set database, and perform transfer learning on the data in the training set database and the test set database; Step 6: Create a query model based on transfer learning and upload it to the cloud. Subsequent query graphics or text are input into the query model. The query model identifies the features of subsequent graphics or text, and queries are performed on the Internet based on the identified features.
2. The efficient query method based on perceptual multimodal big data according to claim 1 is characterized in that: In step 3, the feature selection frequency formula of CountVectorizer is as follows: Among them, IDF is the feature selection probability, m is the number of point cloud data, and g is the number of eigenvalues.
3. The efficient query method based on perceptual multimodal big data according to claim 2 is characterized in that: The ratio of the training set database to the test set database is 1:2.
