Method for retrieving view based on big data

Through data preprocessing, accurate search engines and multimodal fusion technology, the problem of slow big data retrieval speed is solved, the search speed and accuracy are improved, the data processing efficiency and model performance are improved, and interactive and visual search results.

CN119003577AInactive Publication Date: 2024-11-22LANZHOU PUBLIC SECURITY BUREAU
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411083640.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When performing big data retrieval, the search speed of the prior art is slow and cannot effectively process massive data and diverse data types, resulting in inefficient retrieval efficiency.

Method used

Through data preprocessing, the introduction of accurate search engines, partitioned storage and computer application technologies, robot learning and deep learning algorithms, multimodal fusion technology, and holographic projection display, the data classification and retrieval process are optimized, and the search speed and result quality are improved.

Benefits of technology

It improves the speed and accuracy of data retrieval, improves data processing efficiency and model performance, realizes effective integration and joint processing of cross-domain and multimodal data, and provides interactive and visual search results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003577A_ABST
    Figure CN119003577A_ABST
Patent Text Reader

Abstract

The invention discloses a method based on a big data retrieval view, and relates to the technical field of data analysis and visualization, the method comprises the following steps: 1, before a retrieval view is constructed, data in a database needs to be preprocessed, and during preprocessing, the retrieval view is obtained; the accuracy of view construction and the reliability of data processing can be improved through an efficient feature extraction algorithm; secondly, the retrieval speed is increased by introducing an accurate retrieval engine; the method has the advantages that repeated data in a database are cleaned through the data preprocessing technology, the data are classified according to features through a classification algorithm, the classified data are stored in different directory structures, and the data are stored in different directory structures; in the subsequent data retrieval process, the retrieval engine can directly retrieve the data according to the catalog so as to query the needed data, and therefore the retrieval speed is increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and visualization, and in particular to a method for retrieving views based on big data. Background Art

[0002] Modern retrieval technology is gradually developing in the direction of semantic understanding and specific fields. Scientists are making every effort to build "ontology libraries", such as WordNet, HowNet and other ontology dictionaries. Through the ontology library, data is converted into semantic sets, and the semantics of the data is refined to provide semantic-level retrieval. In addition, for the fields of biology, medicine, law, news, blogs, etc., retrieval technologies specifically for a single field have emerged and have developed rapidly. Big data refers to a data set that is so large that it far exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management and analysis. It has four major characteristics: massive data scale, fast data flow, diverse data types and low value density. Big data retrieval, text retrieval, image retrieval, sound retrieval, picture retrieval, etc. are all part of information retrieval. When performing data retrieval in existing technologies, the retrieval engine needs to gradually search the information in the database, which slows down the retrieval speed. For this reason, we propose a method based on big data retrieval view. Summary of the invention

[0003] The object of the present invention is to provide a method for retrieving views based on big data.

[0004] In order to solve the problems raised in the above background technology, the present invention provides the following technical solutions: a method based on big data retrieval view, the method comprising:

[0005] The first step is to preprocess the data in the database before building the retrieval view. During the preprocessing, an efficient feature extraction algorithm can be used to improve the accuracy of view construction and the reliability of data processing.

[0006] The second step is to improve the search speed by introducing accurate search engines;

[0007] The third step is to classify and process the data through partition storage and computer application technology to improve the parallelism and scalability of the query;

[0008] The fourth step is to optimize the classification criteria by minimizing the loss function, and introduce robot learning and deep learning algorithms to improve the processing speed of data classification;

[0009] The fifth step is to use multimodal fusion technology to integrate information from different fields or modalities for cross-domain or multimodal big data retrieval tasks, and to improve the quality of retrieval results through cross-domain retrieval methods.

[0010] Step 6: Display the search box and search content through holographic projection.

[0011] As a further solution of the present invention: the data preprocessing method in the first step includes:

[0012] The data preprocessing method in the first step includes:

[0013] S1. Data cleaning: cleaning and removing erroneous, missing, duplicate or inconsistent data in the database to ensure data quality and accuracy;

[0014] S2. Data conversion, which is used to convert raw data into a form more suitable for analysis and modeling. Data conversion operations include standardization, normalization, encoding, and dimensionality reduction of data to make the data easier to analyze;

[0015] S3, data integration, data integration is the process of integrating data from different data sources to create a unified data view;

[0016] S4. Data classification: determine the labels for data classification. After data cleaning, conversion, integration and reduction, in order to ensure data quality and accuracy, appropriate features can be selected for classification tasks. At the same time, the features of the data are screened, converted and combined, and a classification algorithm is selected to finally classify the data according to the characteristics of the data and the requirements of the classification task.

[0017] S5, data reduction, dimensionality reduction of data, reduce the complexity of data sets, and improve computing efficiency;

[0018] S6. Feature selection, select features in the data and remove irrelevant or redundant features.

[0019] As a further solution of the present invention: in the second step, by introducing an accurate search engine to improve the search speed, the required information can be retrieved from the massive data more quickly.

[0020] As a further solution of the present invention: in the third step, when classifying data, the classified data can be stored in different directory structures, and the data files can be stored in corresponding folders. At the same time, the database establishes indexes through data categories and accelerates data queries through indexes, thereby improving data storage and access efficiency.

[0021] As a further solution of the present invention: In the fourth step, natural language and semantic analysis technology should also be introduced to extract semantic information from text data and establish semantic associations, so as to accurately identify and retrieve relevant data information.

[0022] As a further solution of the present invention: in the fifth step, multi-source data are fused, and data retrieval is performed using a retrieval method that can span different data sources and modalities, combining multi-source data fusion and cross-modal retrieval methods to improve the accuracy and security of retrieval results.

[0023] As a further solution of the present invention: in the sixth step, the search information is projected and displayed to provide visualization and interactivity of the search results, and when querying the results, the visualization technology of reference data can be used to better understand and analyze the query results, while providing an interactive query interface.

[0024] As a further solution of the present invention: the screening formula for data cleaning in S1 is:

[0025] IF(COUNTIF(X:Y,A1)>1,"Duplicate data","Non-duplicate data")

[0026] Among them, COUNTIF is used to count the number of cells that meet specific conditions in the specified range. X:Y is the data range to be filtered, A1 is the position of the current cell, and the formula checks whether there are other cells in the X column that contain the same value as the current cell. If so, "Duplicate Data" is displayed, otherwise "Non-duplicate Data" is displayed. This formula can be used to find duplicate data in the selected data range.

[0027] As a further solution of the present invention: after the data in S1 is filtered, the duplicate data is deleted through ITF=UINH ("duplicate data"), UINH is a delete command, and ITF is an extract command.

[0028] By adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. The present invention cleans duplicate data in the database through data preprocessing technology, and classifies the data according to features through a classification algorithm. The classified data is stored in different directory structures. When performing subsequent data retrieval, the retrieval engine can directly search the data according to the directory to query the required data, thereby improving the retrieval speed;

[0030] 2. The present invention achieves the improvement of data quality and accuracy, the convenience of data analysis and modeling, the establishment of a unified data view, the accuracy of data classification, the improvement of computing efficiency and the optimization of model performance through data preprocessing, thereby improving the efficiency and accuracy of data processing and optimizing model performance, thereby better supporting the construction and analysis of big data retrieval views;

[0031] 3. The present invention uses multimodal fusion technology to integrate different information through cross-domain or multimodal big data retrieval to improve the quality and accuracy of retrieval results. In this case, data of different modalities can be effectively integrated and jointly processed to achieve more comprehensive and accurate retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of a data retrieval method in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] Example 1, please refer to the attached Figure 1 The present invention provides a technical solution: a method for retrieving views based on big data, the method comprising:

[0035] The first step is to preprocess the data in the database before building the retrieval view. During the preprocessing, an efficient feature extraction algorithm can be used to improve the accuracy of view construction and the reliability of data processing.

[0036] The second step is to improve the search speed by introducing accurate search engines;

[0037] The third step is to classify and process the data through partition storage and computer application technology to improve the parallelism and scalability of the query;

[0038] The fourth step is to optimize the classification criteria by minimizing the loss function, and introduce robot learning and deep learning algorithms to improve the processing speed of data classification;

[0039] The fifth step is to use multimodal fusion technology to integrate information from different fields or modalities for cross-domain or multimodal big data retrieval tasks, and to improve the quality of retrieval results through cross-domain retrieval methods.

[0040] Step 6: Display the search box and search content through holographic projection.

[0041] Please see attached Figure 1 , the data preprocessing method in the first step includes:

[0042] S1. Data cleaning: cleaning and removing erroneous, missing, duplicate or inconsistent data in the database to ensure data quality and accuracy;

[0043] S2. Data conversion, which is used to convert raw data into a form more suitable for analysis and modeling. Data conversion operations include standardization, normalization, encoding, and dimensionality reduction of data to make the data easier to analyze;

[0044] S3, data integration, data integration is the process of integrating data from different data sources to create a unified data view;

[0045] S4. Data classification: determine the labels for data classification. After data cleaning, conversion, integration and reduction, in order to ensure data quality and accuracy, appropriate features can be selected for classification tasks. At the same time, the features of the data are screened, converted and combined, and a classification algorithm is selected to finally classify the data according to the characteristics of the data and the requirements of the classification task.

[0046] S5, data reduction, dimensionality reduction of data, reduce the complexity of data sets, and improve computing efficiency;

[0047] S6, feature selection, select features in the data and remove irrelevant or redundant features;

[0048] In this embodiment, the classified data is stored in different directory structures to more effectively organize and manage data and improve the efficiency of data access and retrieval. The data in the database will be updated in real time through the network, and the update time is three days as a cycle. The data in the database will be uniformly cleaned and classified, and stored in different directory structures. When the search engine performs data search, it will first analyze the search content of the search box and associate the search content with the directory. In this way, the search engine can perform targeted searches, thereby greatly improving the accuracy and speed of the search.

[0049] When in use, duplicate data in the database is cleaned through data preprocessing technology, and the data is classified according to characteristics through classification algorithms. The classified data is stored in different directory structures. When performing subsequent data retrieval, the retrieval engine can directly retrieve the data according to the directory to query the required data, thereby improving the retrieval speed.

[0050] Example 2, please refer to the attached Figure 1 The present invention provides a technical solution: a method for retrieving views based on big data, the method comprising:

[0051] The first step is to preprocess the data in the database before building the retrieval view. During the preprocessing, an efficient feature extraction algorithm can be used to improve the accuracy of view construction and the reliability of data processing.

[0052] The second step is to improve the search speed by introducing accurate search engines;

[0053] The third step is to classify and process the data through partition storage and computer application technology to improve the parallelism and scalability of the query;

[0054] The fourth step is to optimize the classification criteria by minimizing the loss function, and introduce robot learning and deep learning algorithms to improve the processing speed of data classification;

[0055] The fifth step is to use multimodal fusion technology to integrate information from different fields or modalities for cross-domain or multimodal big data retrieval tasks, and to improve the quality of retrieval results through cross-domain retrieval methods.

[0056] Step 6: Display the search box and search content through holographic projection.

[0057] Please see attached Figure 1 In the second step, by introducing an accurate search engine to improve the search speed, the required information can be retrieved from massive data more quickly. In the third step, when classifying data, the classified data can be stored in different directory structures, and the data files can be stored in corresponding folders. At the same time, the database establishes indexes through data categories and accelerates data queries through indexes to improve data storage and access efficiency. In the fourth step, natural language and semantic analysis technologies should also be introduced to extract semantic information from text data and establish semantic associations in order to accurately identify and retrieve relevant data information. In the fifth step, multi-source data is fused, and data retrieval is performed using retrieval methods that can span different data sources and modalities. Multi-source data fusion and cross-modal retrieval methods are combined to improve the accuracy and security of retrieval results. In the sixth step, the retrieval information is projected to provide visualization and interactivity of the retrieval results, and the query results can be better understood and analyzed by referencing data visualization technology when querying the results, while providing an interactive query interface.

[0058] In this embodiment, when filtering out duplicate data, the screening formula associates "duplicate data" with the data. In ITF=UINH("duplicate data"), the table data is extracted through the ITF function. After the "duplicate data" appearing in the table is extracted, the data content displaying the "duplicate data" will be deleted.

[0059] When in use, data preprocessing can be used to improve data quality and accuracy, facilitate data analysis and modeling, establish a unified data view, accurately classify data, improve computing efficiency, and optimize model performance, thereby improving data processing efficiency and accuracy and optimizing model performance, thereby better supporting the construction and analysis of big data retrieval views.

[0060] Example 3, please refer to the attached Figure 1 The present invention provides a technical solution: a method for retrieving views based on big data, the method comprising:

[0061] The first step is to preprocess the data in the database before building the retrieval view. During the preprocessing, an efficient feature extraction algorithm can be used to improve the accuracy of view construction and the reliability of data processing.

[0062] The second step is to improve the search speed by introducing accurate search engines;

[0063] The third step is to classify and process the data through partition storage and computer application technology to improve the parallelism and scalability of the query;

[0064] The fourth step is to optimize the classification criteria by minimizing the loss function, and introduce robot learning and deep learning algorithms to improve the processing speed of data classification;

[0065] The fifth step is to use multimodal fusion technology to integrate information from different fields or modalities for cross-domain or multimodal big data retrieval tasks, and to improve the quality of retrieval results through cross-domain retrieval methods.

[0066] Step 6: Display the search box and search content through holographic projection.

[0067] Please see attached Figure 1 , the screening formula for data cleaning in S1 is:

[0068] IF(COUNTIF(X:Y,A1)>1,"Duplicate data","Non-duplicate data")

[0069] Among them, COUNTIF is used to count the number of cells that meet specific conditions in the specified range. X:Y is the data range to be filtered, A1 is the position of the current cell, and the formula checks whether there are other cells in column X that contain the same value as the current cell. If so, "Duplicate Data" is displayed, otherwise "Non-duplicate Data" is displayed. This formula can be used to find duplicate data in the selected data range. After filtering the data in S1, the duplicate data is deleted through ITF=UINH("Duplicate Data"). UINH is the delete command and ITF is the extract command.

[0070] In this embodiment, the search box is displayed through holographic projection. After searching for content through the search box, the search box hides the search content and displays it in the center in an enlarged manner. There may be many types of search results, and the multiple search results are displayed tiled from top to bottom.

[0071] When in use, through cross-domain or multimodal big data retrieval, multimodal fusion technology is used to integrate different information to improve the quality and accuracy of retrieval results. In this case, data of different modalities can be effectively integrated and jointly processed to achieve more comprehensive and accurate retrieval results.

[0072] Working principle:

[0073] The first step is to preprocess the data in the database before building the retrieval view. During the preprocessing, an efficient feature extraction algorithm can be used to improve the accuracy of view construction and the reliability of data processing.

[0074] The second step is to improve the search speed by introducing accurate search engines;

[0075] The third step is to classify and process the data through partition storage and computer application technology to improve the parallelism and scalability of the query;

[0076] The fourth step is to optimize the classification criteria by minimizing the loss function, and introduce robot learning and deep learning algorithms to improve the processing speed of data classification;

[0077] The fifth step is to use multimodal fusion technology to integrate information from different fields or modalities for cross-domain or multimodal big data retrieval tasks, and to improve the quality of retrieval results through cross-domain retrieval methods.

[0078] Step 6: Display the search box and search content through holographic projection.

[0079] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0081] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method based on big data retrieval view, characterized in that: The method comprises: The first step is to preprocess the data in the database before building the retrieval view. During the preprocessing, an efficient feature extraction algorithm can be used to improve the accuracy of view construction and the reliability of data processing. The second step is to improve the search speed by introducing accurate search engines; The third step is to classify and process the data through partition storage and computer application technology to improve the parallelism and scalability of the query; The fourth step is to optimize the classification criteria by minimizing the loss function, and introduce robot learning and deep learning algorithms to improve the processing speed of data classification; Step 5: For cross-domain or multi-modal big data retrieval tasks, multi-modal fusion technology can be used to integrate information from different domains or modalities, while cross-domain retrieval methods can be used to improve the quality of retrieval results; Step 6: Display the search box and search content through holographic projection.

2. The method based on big data retrieval view according to claim 1, characterized in that: The data preprocessing method in the first step includes: S1. Data cleaning: cleaning and removing erroneous, missing, duplicate or inconsistent data in the database to ensure data quality and accuracy; S2. Data conversion, which is used to convert raw data into a form more suitable for analysis and modeling. Data conversion operations include standardization, normalization, encoding, and dimensionality reduction of data to make the data easier to analyze; S3, data integration, data integration is the process of integrating data from different data sources to create a unified data view; S4. Data classification: determine the labels for data classification. After data cleaning, conversion, integration and reduction, in order to ensure data quality and accuracy, appropriate features can be selected for classification tasks. At the same time, the features of the data are screened, converted and combined, and a classification algorithm is selected to finally classify the data according to the characteristics of the data and the requirements of the classification task. S5, data reduction, dimensionality reduction of data, reduce the complexity of data sets, and improve computing efficiency; S6. Feature selection, select features in the data and remove irrelevant or redundant features.

3. The method based on big data retrieval view according to claim 1, characterized in that: In the second step, by introducing an accurate search engine to improve the search speed, the required information can be retrieved from the massive data more quickly.

4. The method based on big data retrieval view according to claim 1, characterized in that: In the third step, when classifying data, the classified data can be stored in different directory structures, and the data files can be stored in corresponding folders. At the same time, the database establishes indexes through data categories and accelerates data queries through indexes, thereby improving data storage and access efficiency.

5. The method based on big data retrieval view according to claim 1, characterized in that: In the fourth step, natural language and semantic analysis technology should also be introduced to extract semantic information from text data and establish semantic associations in order to accurately identify and retrieve relevant data information.

6. The method based on big data retrieval view according to claim 1, characterized in that: In the fifth step, multi-source data are fused, and data retrieval is performed using a retrieval method that can span different data sources and modalities. Multi-source data fusion and cross-modal retrieval methods are combined to improve the accuracy and security of retrieval results.

7. The method based on big data retrieval view according to claim 1, characterized in that: In the sixth step, the search information is projected and displayed to provide visualization and interactivity of the search results. When querying the results, the visualization technology of reference data can be used to better understand and analyze the query results, while providing an interactive query interface.

8. The method based on big data retrieval view according to claim 1, characterized in that: The screening formula for data cleaning in S1 is: IF(COUNTIF(X:Y,A1)>1,"Duplicate data","Non-duplicate data") Among them, COUNTIF is used to count the number of cells that meet specific conditions in the specified range. X:Y is the data range to be filtered, A1 is the position of the current cell, and the formula checks whether there are other cells in the X column that contain the same value as the current cell. If so, "Duplicate Data" is displayed, otherwise "Non-duplicate Data" is displayed. This formula can be used to find duplicate data in the selected data range.

9. The method based on big data retrieval view according to claim 1, characterized in that: After the data in S1 is screened, duplicate data is deleted through ITF=UINH ("duplicate data"), where UINH is a delete command and ITF is an extract command.