Data storage for list-based data search

Abstract

This invention relates to a method for storing data (114, 115, 116) in a data storage device (104), the method comprising: a data processing search (DVS) system (102) receiving (202) raw data (112) or a raw data access address; multiple different parsers parsing (204) the raw data to determine data objects, each with one or more data values ​​and an object ID for each data object, wherein at least some data values ​​are assigned semantic concepts; the DVS system automatically importing (206) the parsing results; and the DVS system presenting all parsing results as data values ​​without redundancy. Lists (114, 115, 116) are automatically stored (208) in the data storage. These non-redundant lists include: one or more concept lists (116), each concept list representing a semantic concept, the non-redundant list selectively including imported data values ​​assigned to the semantic concepts in the concept list during parsing, and a non-concept list (115), the non-concept list selectively including imported data values ​​not assigned semantic concepts during parsing; the DVS system provides (210) non-redundant lists (114, 115, 116) to answer search queries and / or perform data analysis.

Description

Technical Field

[0001] This invention relates to a method and system for processing stored data in order to search for the data. Background Technology

[0002] Various database management systems (DBMS) for storing, managing, and efficiently processing data are disclosed in the prior art. The primary task of a DBMS is to efficiently and consistently store large amounts of data persistently, providing users and applications with the necessary subsets in different representations as needed. In databases managed by traditional DBMSs, the foundation for constructing data and its relationships is the database model specified by the DBMS vendor. Based on the database model, the database schema must adapt to certain structured options. Currently known database models include hierarchical models, network-like models, relational models (organized in a table format), object-oriented models, document-oriented models, and hybrid forms of the above. Furthermore, DBMSs are traditionally differentiated based on whether they are optimized for efficiently answering many short queries (OLTP) or lengthy evaluations (OLAP).

[0003] In the case of a conventional DBMS, when choosing a DBMS and defining the internal structure of the database managed by the DBMS (especially the size, number, and internal references of the database tables, and the selection of table column types for creating indexes), database developers are required to handle the content (information content) of the data to be managed and the types of queries that the DBMS may need to process.

[0004] The problem is that both of these aspects change over time and are often unknown or only partially known when the database is built. In practice, it's common to need to verify the relevance of other or additional aspects of the data stored in the database, requiring new queries that couldn't have been considered when the database was initially built. For example, if the database contains medical data, and new medical knowledge emerges after its creation—that several symptom combinations can predict a specific diagnosis—the database may indeed contain these symptoms, but the symptoms themselves may be distributed across different tables and / or indexed in an inefficient way, consuming significant memory, CPU resources, and time. Furthermore, due to the numerous dependencies between the data in the database and the client systems, subsequent adjustments to the defined database structure are extremely time-consuming, error-prone, and often haphazard.

[0005] Furthermore, over time, the problem often arises where more data sources need to be integrated into the database, but their internal structures do not match the data model chosen when the database was built. For example, if a database with a relational data model is chosen and other hierarchical data needs to be integrated, it is usually not possible to store the other data in the database in this way, even if the other data may be conceptually related to the existing data in the database. Even if the other data can be stored in a data model that is inherently unsuitable, it is impossible to achieve general querying and semantic integration between the existing data and the other data, at least not without a fundamental redesign of the data organization in the database.

[0006] Therefore, existing DBMSs often suffer from inflexible structure, poor scalability, and / or poor performance, especially when handling complex queries involving a large number of data objects with many different attributes (keys) and corresponding values. If other data with different structures need to be stored in the database in the future, it is usually impossible to query and analyze existing and new data in a reasonable, high-performance, and resource-efficient manner. This situation is somewhat special, but not limited to the context of the "Internet of Things," because various objects and sensors are used to record highly heterogeneous data in terms of content and structure, and the composition and properties of the data often change over time. Summary of the Invention

[0007] The purpose of this invention is to propose an improved method and system for storing data in a way that allows for efficient data searching.

[0008] To achieve the above objectives, the present invention provides the features of the independent claims. Specific embodiments of the invention are described in the dependent claims. The embodiments presented below can be freely combined as long as they are not mutually exclusive.

[0009] On one hand, the present invention relates to a method for storing data in a data storage device. The method includes:

[0010] - The Data Processing Search System (DVS system) receives raw data or raw data access addresses, and the raw data has different structures;

[0011] - Multiple different parsers parse the raw data to determine the data objects, each with one or more data values ​​and the object ID of each data object, with at least some data values ​​assigned semantic concepts;

[0012] - The DVS system automatically imports the parsing results;

[0013] The DVS system automatically stores all parsed results in a list of non-redundant data values ​​in the data storage device. This non-redundant list includes:

[0014] • One or more concept lists, each representing a semantic concept. The list without redundancy selectively includes the imported data values ​​assigned to the semantic concepts in the concept list during parsing. Each data value in the concept list is assigned the object ID of all data objects containing that data value. The data values ​​contained in the list represent the semantic concepts in the concept list.

[0015] • No-concept list: The no-concept list selectively includes imported data values ​​that were not assigned semantic concepts during parsing. Each data value in the no-concept list is assigned the object ID of all data objects that contain that data value. The data values ​​contained in the data objects were not assigned semantic concepts during parsing.

[0016] - The DVS system provides a non-redundant list to answer search queries and / or perform data analysis.

[0017] This is advantageous because it allows for the transformation of highly heterogeneous raw data in terms of structure and content into a universal structure, enabling the rapid and efficient processing of large amounts of data (intersections, unions, differences, or symmetric differences of object IDs from elements in multiple lists). It allows for continuous expansion in terms of content (by adding more preferred, automatically generated lists of concepts) and supported query and analysis processes without fundamentally altering the data structure within the data store. These lists are non-redundant, so each data value is included only once in a list. If it appears in several data objects within the raw data, this data value, appearing only once in a list, is stored as a link to those object IDs. This allows for rapid and efficient processing even of large databases, and even on computer systems with limited main memory and / or CPU capacity. Such a data representation can lead to significant data compression effects, especially when the raw data contains certain data values ​​multiple times (e.g., in the case of text representing a finite number of existing word combinations or proteins representing a finite number of amino acid sequences). For example, object IDs can each represent a dataset (e.g., protein sequence ID, raw data file URL, natural language text or statement, row of an Excel file, etc.), and data values ​​are values ​​that appear in that dataset (e.g., words, bit strings, numbers, etc.).

[0018] Assigning semantic concepts to data values ​​is often a part of the parsing process.

[0019] According to an embodiment of the present invention, the DVS system stores imported data values ​​in a list of non-redundant data values ​​based on the semantic concept of assigning values ​​to imported data values ​​(provided that this can be determined at parsing time or otherwise specified), wherein the storage and distribution of data values ​​in the list are independent of the original dependency relationship between the data values ​​and the data objects.

[0020] Therefore, the advantage of this invention lies in its significant flexibility in integrating large amounts of heterogeneous data with related structure and content. This results in superior performance during data analysis and search because, structurally, importing data values ​​into the data storage solves the problem of initially assigning them to data objects. Data values ​​are only associated with objects through object IDs, without affecting the data structure managed by the DVS system. Regardless of whether the original data is provided in a hierarchical, tabular, or XML file format, and regardless of whether data values ​​are extensively specified in the original data (e.g., via key-value fields) or dynamically acquired by one of the parsers during analysis (e.g., through pattern recognition of image or audio data), data values ​​are always linked to a non-redundant list stored along with the object IDs containing the data values. During this storage, the semantic concept represented by the data value is considered, specifically by selectively storing the data value in a list of concepts that precisely represent that concept. If the data value already exists in this list, only the corresponding set of object IDs is added. If the semantic concept behind the data value cannot be determined when parsing the original data, the data value is stored in a "no concept" list, or if the data value already exists here, only the corresponding entry is expanded for the object ID.

[0021] The import process can refer to the process of parsing the structure of all data objects, in which all data values ​​extracted from all data objects of the original data are stored in a list of non-redundant data values ​​based on the semantic concepts assigned to them.

[0022] For example, data sources such as JSON files, XML files, database tables, media files, or user input via a user interface (e.g., a graphical user interface) can be provided to the DVS system. This can be achieved by the DVS system receiving the raw data (original or copy), for example, via a network or from local data storage or other sources, and initially storing it in a copy within its own document storage. The advantage of this is that the original remains available, and can be used to display search or analysis results if necessary, or for subsequent parsing and analysis. However, the raw data may not be copied and stored in the DVS system's document storage; instead, the DVS system may only have (permanent or temporary) read access to the raw data in order to process and parse it. In this case, a DVS system receiving the raw data access address is sufficient.

[0023] The advantage of this invention's embodiments lies in its ability to query data objects described by a large number of attributes (values) for different semantic concepts (keys) (even thousands per object), with extremely short query times for any combination of different key values, and without adhering to specifications dependent on the original structure of the data objects (e.g., this is the case in index-based relational DBMSs regarding table and index structures in the database). In index-based systems, indexes for all possible combinations of keys must be available. Traditionally, the set of indexes in index-based DBMSs grows with the factorial of the keys! Especially for a large number of different object types with many different semantic concepts, the number of indexes required increases with any possible combination of key-related search conditions and key factorials! According to the embodiments of this invention, however, there is no need to generate and use indexes (in the sense of a searchable data structure generated from outside the data values, such as a B-tree). According to the embodiments of this invention, a non-redundant list of concepts corresponds to each semantic concept, where each data value of that concept (e.g., distinct from a dataset-based relational DBMS table) appears only once. According to the embodiments of this invention, searches and / or analyses can be performed without creating a suitable index structure for the expected search query. Specifically, a search can be performed directly in a list without redundancy, preferably a list of concepts that correspond to the semantic concepts referred to by the search criteria (and, if necessary, a list without concepts and / or a global list).

[0024] According to embodiments of the present invention, the data storage device is a volatile or non-volatile data storage device. For example, the data storage device may be a hard disk (HDD), a mass storage device (SSD), or main memory (a memory-based management list).

[0025] According to embodiments of the present invention, at least some of the original data exists in or is received in a multi-data-structure format. The multi-data-structure format may particularly include a mixture of two or more of the following data structures:

[0026] -XML file;

[0027] - JSON file;

[0028] -Text file;

[0029] -CSV file;

[0030] - Database tables;

[0031] -Object tree;

[0032] -Media files, especially video files, audio files, and / or image files;

[0033] - Data entered via the GUI;

[0034] - Streaming data.

[0035] Streaming data is data continuously generated by a source system and sent in packets and / or queues (according to the FiFo principle). Typically, this type of data is processed "in real time" by a streaming framework. The streaming framework accepts this data stream, processes the information in main memory, and then writes it to a data storage device (e.g., a hard disk (HDD) or mass storage such as a solid-state drive (SSD)). According to embodiments of the invention, the DVS system can also be designed to process raw streaming data in real time. For example, streaming data could be log data from an ERP system, e-commerce events (views, orders, shopping carts), tracking events on a mobile application, geolocation data from a web application, or usage data from a specific product.

[0036] However, the above list should not be interpreted as an exhaustive list.

[0037] According to an embodiment of the invention, at least some (generally most) of the non-redundant lists contain data values ​​from the original data that are contained in two or more different data structures. Thus, the question of which data structures the data values ​​appear in within the original data has no impact on the list structure managed by the DVS system. This is advantageous because the system is independent of the structure of the file and database containing the original data.

[0038] The raw data can be of varying content and structure. This is advantageous because the DVS system allows for the integration of large amounts of heterogeneous data from numerous independent sources into a common data structure (a list of non-redundant data values). This significantly compresses the dataset while semantically integrating it, at least when the raw data from two or more different sources have at least one data value representing the same semantic concept, which is then identified during parsing. In this case, the data value for that concept is stored as a reference linked to various data objects in the list of the same concept.

[0039] Another significant advantage is that it's not necessary to fully integrate data objects from different sources semantically. For example, if the original data is a protein or gene sequence, and during the parsing process, certain peptide sequences or genomic markers representing the concept of "predictors of diagnosis X" are identified in the original data from other data sources, it doesn't matter if the parser used cannot identify and extract all the semantic concepts that might exist in the original data from the data sources. This allows for the integration of data from different heterogeneous sources without requiring a complete "understanding" or a complete "data model." However, if, during the parsing of original data from different data sources, at least one or more data values ​​representing certain semantic concepts are found, these data values ​​can be categorized into the appropriate lists for integration.

[0040] Even if the semantic concepts are unknown, heterogeneous data can still be integrated based on relevance: for example, if certain genomic markers and diagnoses are frequently mentioned in the literature, this can result in a large overlap of object IDs for data objects that contain both genomic markers and diagnoses (expected to be higher than statistically likely given the frequency of individual data values). If the biological function and related semantic concepts behind the marker are unknown, the marker will be stored as a data value in a list of no-concepts, allowing it to be considered when processing search queries or data analysis (such as relevance analysis), completely independent of whether the genomic marker is provided as an XML file, text file, or any other format.

[0041] The embodiments of this invention also possess the advantage of high flexibility and scalability: since the structure of data objects in the original data, as well as the type and number of data object attributes, have no impact on the data structure managed by the DVS system (these data are always stored in a non-redundant list format), additional original data can be imported and integrated into the new structure at any time. If the structure of the original data is unknown in advance, only a corresponding parser needs to be developed to process the original data. Where possible, semantic concepts are assigned to the data values ​​contained therein, and based on the existence and type of the assigned semantic concepts, the data values ​​and the object IDs of the relevant data objects, or at least the object IDs, are stored together in either a list without concepts or a list with concepts.

[0042] According to an embodiment of the present invention, the above method includes: the DVS system or another DVS system performing search queries and / or analysis.

[0043] According to embodiments of the present invention, search queries and / or data analysis are performed, particularly without accessing the original data.

[0044] According to an embodiment of the present invention, the DVS system is configured to perform analysis and / or search only based on non-redundant lists or on data derived from these lists by the DVS system, in particular not to analyze and / or search the original data or the data objects contained therein or the datasets representing these data objects.

[0045] The advantage of this approach is that search and analysis are independent of the structure of the original data. It can compress terabyte-scale amounts of raw data to a size that can be quickly processed by standard computing, and sometimes even single-board calculators (such as Raspberry Pi), while also enabling the searching of complex queries. The compression is based on the principle that each data value appears only once in each concept list and no-concept list. If several different semantic concepts (surname, color, metal type) are assigned to a data value, the data value (e.g., "Silber") may appear in more than two concept lists. However, the value "Silber" appears only once in each list. If the data value "Silber" is found in other data objects, where "Silber" represents the concept "Metall," then the list element "Silber" is selectively assigned an additional object ID to the data object in the list representing the concept "Metalltyp." If, when parsing some raw data, the parser cannot identify whether the data value "Silber" represents a color, metal, surname, or other semantic concept not previously listed, then this data value "Silber" is saved to the no-concept list. Therefore, the data value "silver" may appear thousands of times in the original data, but only once or a few times in the list that the DVS system is working on. Object IDs are preferably stored in numerical form, occupying very little storage space and allowing for fast set operations. This enables complex search queries, for example, to be performed on all data objects containing the product type "wire" and the metal type "silver," which can be answered by calculating the intersection of object IDs based on the set of object IDs assigned to "wire" in the "product type" concept list and "silver" in the "metal type" concept list. Data analysis no longer needs the structure of the data objects contained in the original data.

[0046] According to an embodiment of the present invention, the data storage managed by the DVS system does not contain data objects (also known as datasets).

[0047] According to other implementations, the DVS system also manages a document store containing all or at least some of the data structures containing the raw data. However, this raw data is also not used for processing search queries or data analysis, but for other purposes. For example, it may be advantageous to store a copy of the raw data in the DVS system's document store so that the parsing process can be performed again on the raw data at a later time when an additional and / or better parser becomes available, in order to identify and extract additional data values ​​and / or semantic concepts assigned to them that the previously used parser could not recognize or parse. This reduces network traffic because there is no need to retransmit the raw data stored in the networked system when a new or better parser becomes available.

[0048] According to an embodiment of the present invention, the redundancy-free list further includes a global list. The global list is a redundancy-free list of all imported data values. Each data value in the global list is assigned one or more pointers, each pointer pointing to an element in one of the concept lists or an element in the no-concept list, the element containing the same data value as the data value. The DVS system is configured to perform at least analysis or query searches on the global list, which is used to identify and / or process data values ​​representing different semantic concepts in different data objects for concept processing.

[0049] Preferably, data values ​​in the global list are not assigned object IDs. For example, the global list will only contain the data value "silver" once. Although this data value in the global list is not directly assigned an object ID, it is assigned one or more pointers to a list of concepts representing the semantic concepts assigned to the data value "silver" in at least one data object. In this example, the data value "silver" in the global list will be assigned the concept lists "surname," "metal type," and "color." If the data value "silver" is stored in a list without concepts, then the data value "silver" will also be assigned to a list without concepts.

[0050] The advantage of using a global list is that it allows for immediate determination of whether each data value represents a different semantic concept based on the context. If so, it determines the number and types of semantic concepts. Preferably, only data values ​​in each concept list or no-concept list are directly assigned object IDs. Therefore, it is easy to select data objects that contain the data value "silver" according to a specific concept and distinguish them from data objects that contain the word "silver" according to another concept.

[0051] According to one embodiment, the DVS system is designed to provide a GUI for inputting search terms. During search term input, the DVS system searches a global list to determine if the input term corresponds to a data value in the global list. The DVS system suggests one or more specific data values ​​that are at least partially identical or similar to the entered search term. For example, the DVS system may display specific data values ​​to the user in a context menu or autocomplete list, making them available for selection. When the user selects one of the suggested data values, the DVS system determines the number and type of semantic concepts (and corresponding concept lists) referenced by that data value in the global list. These determined semantic concepts are also displayed on the GUI, through which the user can select one or more semantic concepts. For example, when the user enters the search term "silver," several known concepts for that search term (surname, metal type, color) are also provided. The user can select these concepts, for example, through GUI elements such as "checkboxes," "drop-down lists," and "radio buttons," thereby limiting the search query to specific semantic concepts. For example, selecting the concept "metal type" limits the search to data objects that contain the data value "silver" as a metal identifier. The object IDs of these data objects are stored in a list of "metal types" concepts and associated with the "silver" list element.

[0052] According to embodiments of the present invention, at least some data values ​​are extracted from data structure fields containing the original data. These fields are defined by the data structure. These fields include one or more concept-related fields and / or one or more concept-independent fields.

[0053] For example, a field can be a cell in an Excel file, an element in a database table, or a specific XML element within an XML object tree. It corresponds to a predefined container within a data structure used to store one or more data values. Conceptually relevant fields are fields assigned field identifiers that represent semantic concepts. Conceptually irrelevant fields are fields without assigned semantic meaning. Typically, at least some data values ​​from the original data are stored in concept-relevant fields and / or some other data values ​​from the original data are stored in concept-irrelevant fields. For example, if an Excel spreadsheet is parsed and its rows and columns do not contain identifiers representing the semantic concepts of the data contained in the table, these fields can be considered concept-irrelevant fields. However, if field identifiers such as "surname," "first name," "date of birth," "date of production," and "weight" exist, then they are concept-relevant fields.

[0054] Parsing and importing data values ​​stored in concept-related fields can be advantageous because it allows new semantic concepts to be automatically "learned" and integrated into the DVS system. Certain parser types can also achieve excellent learning performance: by defining a parser that interprets the data values ​​in the first row of a table into the semantic concepts that the data values ​​in the same column below those cells should be assigned, this parser can be used to import new semantic concepts and new data values ​​from any Excel spreadsheet that follows a scheme where the semantic concepts of each column are contained in the first row (header row). If the DVS system recognizes a previously unknown semantic concept "last name" (i.e., a list of "last name" concepts does not exist) when importing data values ​​from the "last name" column, the DVS system will automatically create such a list, storing all data values ​​for that column without redundancy.

[0055] According to an embodiment of the present invention, the DVS system stores the data values ​​extracted from concept-related fields in a concept list representing the semantic concepts of the field identifiers of the concept-related fields. According to an embodiment of the present invention, if the parser used does not recognize the semantic concepts of the data values, the DVS system stores the data values ​​extracted from concept-independent fields solely in a no-concept list.

[0056] According to embodiments of the present invention, at least some data values ​​are imported by semantic parsers that identify the imported data values ​​and assign semantic concepts to them based on data analysis, particularly image analysis, audio signal analysis, statistical analysis, classification, machine learning, and / or pattern recognition. According to one embodiment, the multiple parsers used include parsers that classify audio data and convert it into text, then decompose it into words used as data values. Alternatively or supplementarily, the parsers may evaluate the audio data to identify music genres and treat the audio data as a whole as data values, assigning it an automatically determined music genre (e.g., rock, techno, classical) as a semantic concept.

[0057] Even if no field identifiers indicate a semantic concept for the data value, the semantic concept can still be identified during the parsing process. For example, the raw data could be a digital image, and the parser could be image analysis software capable of identifying specific objects (people, animals, plants, etc.). The image or image fragment that identifies the specific object obtains the object ID and the image (or the pixel data of the corresponding image fragment, or only the name of the identified object, "person," "dog," "cat," etc.) as the parsed "data value" stored in the corresponding concept list ("person" or "dog," etc.). Whether a data value can be identified and extracted from the raw data, and whether a semantic concept can be assigned to that data value, can depend on the field identifiers and / or the parser used.

[0058] If the list of non-redundant data values ​​still does not contain a list of concepts for dynamically identified new semantic concepts (new music genres, new object types in image data), the DVS system preferably automatically creates a new list of concepts for that semantic concept.

[0059] According to an embodiment of the present invention, the above method further includes:

[0060] - After importing and storing data values, an additional parser is provided, which is configured to identify and import data values ​​that are given at least one new semantic concept, which is a concept not represented by any list of concepts contained in the current data store;

[0061] - An additional parser processes the raw data and extracts one or more new data values ​​from the raw data that give rise to at least one new concept;

[0062] The DVS system compares at least one new semantic concept identified by another parser with a list of concepts in the data storage, and for each new semantic concept, the other parser extracts at least one data value, automatically generating and storing a list of new concepts.

[0063] - Automatically store the data values ​​extracted from the original data by another parser in at least one new concept list, representing the new concept list to which the other parser assigns a new semantic concept to the data value.

[0064] - Preferably, the no-concept list and / or global list are also updated in such a way that the object IDs of data objects to which data values ​​are assigned, where these data objects contain data values ​​that a new parser can assign to a new concept for the first time, are removed from the no-concept list. These object IDs are then assigned to this data value in the new concept list. For this data value, the no-concept list retains the object IDs of those data objects that contain that data value but for which other parsers cannot assign a concept.

[0065] This also allows the same raw data to be evaluated subsequently using the new and improved parser, in which case the extracted knowledge can be seamlessly integrated into the existing list structure, making it searchable.

[0066] According to an embodiment of the present invention, the above method further includes:

[0067] - Receive search queries that identify data objects that satisfy one or more concept-related search conditions and / or one or more concept-independent search conditions. Concept-related search conditions are search conditions that are assigned semantic concept identifiers, and concept-independent search conditions are search conditions that are not assigned semantic concepts.

[0068] - Search the list of no-concepts for one or more data values ​​that satisfy each received concept-independent search condition; for example, the search could be for the same or similar data values ​​related to the search term; for example, regular expressions could be used for similarity searches; and / or

[0069] - For each received concept-related search condition, selectively search the list of concepts representing the semantic concepts of that search condition for one or more data values ​​that satisfy the concept-related search condition;

[0070] - Returns the object ID or a subset of these object IDs that the DVS system determines in response to a search query to search the global list and / or at least one list of concepts.

[0071] By powerfully compressing data values ​​and automatically splitting them into different lists based on semantic concepts, massive amounts of raw data can be processed quickly, even on low-performance computers, transforming the data values ​​contained therein into a list structure.

[0072] According to a preferred embodiment of the present invention, the data values ​​in each list are sorted, that is, arranged according to a sequential relationship (for example, for data values ​​consisting of alphanumeric strings, arranged in alphabetical order). This can speed up the search for a specific data value in the list, because if it is determined that the list does not contain the data value after traversing the data values ​​based on the sequential relationship, the search can be stopped.

[0073] According to embodiments of the present invention, using a non-redundant list for query searching and / or data analysis includes performing set operations on a set of object IDs assigned data values ​​to two or more non-redundant lists, particularly intersection, union, difference, or symmetric difference operations.

[0074] For example, one or more data values ​​(“matching values”) that are exactly the same as or most similar to the search terms in a search query can be identified from multiple concept lists and / or no-concept lists. By applying a union operation to the set of object IDs assigned to these matching values ​​for individual concept lists in these concept lists, the union of all data objects containing matching values ​​for that concept list is calculated. By applying an intersection operation to the set of object IDs obtained in the union step of the different concept lists, the intersection of data objects for each concept list that contain at least one matching value for the search terms of the query is calculated. This intersection can be returned as a search query result and / or used for further analysis.

[0075] According to embodiments of the present invention, the data values ​​of the no-concept list and / or one or more concept lists are each stored as links to one or more time records, all of which contain the transaction time when the data values ​​were initially generated or the time when the DVS system imported the data values. The method also includes the DVS system considering time records to respond to search queries and / or perform data analysis.

[0076] For example, the data value "Frankfurt" in the concept list "Firmensitz" can be assigned the date "since March 13, 1978" and the object ID of the business register No. 238 for the company "Gelb AG". Similarly, the data value "Berlin" in the concept list "Firmensitz" can be assigned the date "since July 16, 2012" and the object ID of the business register No. 46474 for the same company "Gelb AG". By querying all the "Firmensitz" data values ​​for the company "Gelb AG", the chronological order of the company's registered address changes can be reconstructed, as the data objects from which the data values ​​are extracted and their object IDs are linked to this time record.

[0077] According to an embodiment of the present invention, at least some data objects have relationships with other data objects. These relationships can be, for example, "is Bestandteil von (belongs to)", "beinhaltet (contains)", etc. Relations such as "zu Klasse (genus)," "aktiviert (activation)," "inhibiert (blocking)," and "ist_verbaubar_in (matching)" exist. These relations may be explicitly or implicitly contained in the original data and can be extracted during the parsing process. However, the DVS system may also extract relations only later when performing data processing operations on the original data and / or data values ​​in a stored, non-redundant list.

[0078] The relationship between each extracted data object (“first data object”) and another data object (“second data object”) is extracted as a combination (specifically, a concatenation) of the relationship type (i.e., a relationship identifier, such as “is_verbaubar_in”) and the object ID of the second data object. Each extracted combination is stored as one of the data values ​​in a list of no concepts and / or at least one list of concepts, and each extracted combination in the list of no concepts and / or at least one list of concepts is assigned the object ID of the first data object in which the specified relationship exists.

[0079] For example, this extraction and storage of object relationships can be used to formulate "open" search queries, extracting the context of a specific object, including the set of objects referencing that search object and the type of relationship.

[0080] According to an embodiment of the present invention, the object ID of the data object determined during parsing is also regarded as a data value and stored in, for example, a list of "object ID" concepts (optionally, additionally stored in a global list).

[0081] For example, the object ID of a data object can be extracted as a data value and stored in a list of "object identifiers" concepts. A specific object ID can be searched in the object identifier concept list (search object ID) to find a data value that is exactly the same as the search object ID. Then, the object IDs of all data objects in the object identifier concept list that are assigned to that determined data value are determined. This includes not only data objects that are exactly the same as the search object, but also all data objects that are related to the search object, as they also contain the object ID of the search object.

[0082] In addition to searching for "identical" matches as described above, a global list can also be searched using regular expressions that include the object ID of the search object. For example, a global list could contain data values ​​for many mapping relationships, such as "ist_verbaubar_in_Motor_XL-3000", "ist_leistungsequivalent_zu_Motor_XL-3000", and "wird_aktiviert_von_p53_Gen", where the components "Motor_XL-3000" and "p53_Gen" represent object IDs. By searching using regular expressions that specify only the object ID as the search object ID and keep the relation type identifier open, all data values ​​in the global list that contain the search object ID as a component of the data value can be identified first—that is, all data values ​​referencing the relation type of the search object. The global list data values ​​thus identified reference the corresponding context list, such as the context lists for the semantic concepts "ist_verbaubar_in" and "ist_leistungsequivalent_zu". The concept list "ist_verbaubar_in" can contain the data value "ist_verbaubar_in_Motor_XL-3000" assigned the first set of object IDs. The concept list "ist_leistungsequivalent_zu" can contain the data value "ist_leistungsequivalent_zu_Motor_XL-3000" assigned the second set of object IDs. Thus, the combination of the first and second set of object IDs produces all data objects for the search object "Motor_XL_3000" for either "verbaubar in (matched)" or "leistungsequivalent zu (power equal to)".

[0083] According to an embodiment of the present invention, the above method includes:

[0084] - Receive a search query or analysis command that identifies all data objects related to the search data object, the request or command containing the search object ID;

[0085] - Search the global list to determine all data values ​​composed of the relation type and the object ID that is the same as the search object ID;

[0086] - Identify one or more lists of concepts related to the data values ​​determined in the global list;

[0087] - Search one or more identified lists of concepts to determine data values ​​composed of a relation type and an object ID that is the same as the search object ID;

[0088] - Returns the object IDs of one or more concept lists that assign the determined data values, i.e., the object IDs of the data objects that are related to the search object.

[0089] This allows for highly flexible, scalable, and high-performance execution of highly complex search queries and similarity searches (using regular expressions for fuzzy searches) without the need to create database indexes in the form of B-trees, which would otherwise require adjustments when integrating new data object types or search queries. Furthermore, for each search object, it is possible to quickly identify which relation types exist about the search object, how many other objects reference or are connected to it through one or more relation types.

[0090] This is particularly advantageous in the context of Industry 4.0 applications or in situations involving the ordering and / or configuration of highly complex components (aircraft turbines and engines): complex components or system components may consist of thousands of parts, which in turn may be composed of sub-components. If an automotive or aircraft manufacturer's database contains a large number of components, this type of relational query can quickly determine which data objects, for example, have the relation "ist-Bestandteil-von-Motor-XM-300049BT".

[0091] According to an embodiment of the present invention, the above method further includes:

[0092] The DVS system provides a mapping table that assigns one or more identifiers to each data value in the no-concept list and each data value in the concept list;

[0093] - The DVS system generates a list of no concepts and a confused copy of each concept list. The data values ​​in the confused copies are all replaced with the identifiers assigned to those data values ​​in the mapping table.

[0094] The DVS system uses obfuscated lists for search queries and / or analysis.

[0095] The advantage of this approach lies in enhanced data security. For example, the mapping table and the obfuscated list can be provided on different computer systems and / or stored and used in separate IT security environments. If an unauthorized third party gains access to the obfuscated list, the list's contents cannot be identified in the absence of the mapping table. Accessing the mapping table itself is harmless, as it does not contain any object IDs that would allow the reconstruction of data values ​​assigned to objects. An unauthorized third party would have to compromise two separate computer systems or two separate security systems to reconstruct the original list. This provides a particularly secure method for data management and searching.

[0096] According to embodiments of the invention, the identifier is a value whose length and / or type depends on the processor architecture of the computer system used for searching and / or analysis; that is, a value selected based on these factors. In particular, the length and / or type of the identifier are chosen to enable the processor architecture to process these identifiers at high speed. For example, the length of the identifier may correspond to the processing width of the processor architecture (e.g., 32 bits for a 32-bit architecture, 64 bits for a 64-bit architecture). If the processor architecture can process numerical values ​​particularly efficiently, the identifier may consist of numerical values. If the processor architecture can process other types of values ​​(e.g., symbols) particularly efficiently, the identifier may consist of symbols.

[0097] According to an embodiment of the present invention, when performing a search or analysis based on a mapping table, a search value identifier is determined, which is an identifier assigned to the search value in the mapping table.

[0098] According to embodiments of the invention, all identifiers have a fixed length, particularly the same length, which is preferably chosen such that each identifier is perfectly suited to the working register of the ALU of at least one processor performing the method according to embodiments of the invention. Thus, according to embodiments of the invention, identifiers are compared with each other and / or with search value identifiers within a single working cycle of the ALU.

[0099] This is advantageous because replacing data values ​​with one or more identifiers not only improves data content protection against unauthorized access, but also increases the efficiency of data processing during search queries and data analysis.

[0100] According to an embodiment of the present invention, the above method further includes:

[0101] - List of encryption concepts and list of no-concepts in the DVS system, or list of encryption obfuscated concepts and list of obfuscated no-concepts in the DVS system;

[0102] - In response to receiving a search query or data analysis command, identify an encrypted list that needs to be processed to handle the search query or data analysis command;

[0103] - Verify the sender of the search query with the DVS system, where the DVS system checks whether the sender is authorized to access the list content for each identified list;

[0104] - The DVS system selectively generates decrypted copies of the identified list, which are verified to be accessible to the sender.

[0105] - Perform search queries or data analysis on the decrypted copy of the list;

[0106] - After a search query or data analysis, the DVS system deletes the decrypted copy of the list.

[0107] The advantage of this is that the lists generated and managed by the DVS system are essentially unreadable due to encryption. Preferably, the encryption keys required for decryption are securely stored, for example, on an HSM or in a relatively secure environment. This means that unauthorized third parties who obtain the list or a copy of the list cannot read the data contained therein. Only in response to search queries or analysis commands are some parts of the list temporarily decrypted to answer the query. This provides a flexible, scalable, resource-efficient, and secure method for managing and searching data.

[0108] According to a preferred embodiment, the DVS system uses different encryption keys to encrypt each list. This further enhances security.

[0109] According to embodiments of the present invention, at least one list of concepts represents genomic markers of multiple individuals of a species or subspecies, and at least one additional list of concepts represents metabolic or phenotypic characteristics of these individuals. The DVS system performs data analysis by performing set operations on these lists. The data analysis involves both correlation analysis of genomic markers and correlation analysis of metabolic or phenotypic characteristics. This provides a highly efficient and novel method for conducting so-called "genome-wide association studies (GWA)."

[0110] According to other implementations, at least some data values ​​of the global list and / or at least one concept list specify an "ist-Bestandteil-von" relationship with other objects, including the object ID of the other object, and at least one additional concept list represents the metabolic or phenotypic characteristics of these individuals. The DVS system performs search queries, with search criteria comprising a combination of the relationship type "ist-Bestandteil-von" and object IDs, where the object IDs represent a machine or vehicle or multiple components of that machine or vehicle.

[0111] According to other implementations, at least some data values ​​of the global list and / or at least one concept list are natural language words. The DVS system performs data analysis by performing set operations on these lists. The data analysis involves word correlation analysis to create a predictive model of the natural language.

[0112] On the other hand, the present invention relates to a volatile or non-volatile storage medium storing computer-readable instructions designed to cause a processor to perform a method of storing data in a data memory according to any one of the preceding claims.

[0113] According to one implementation scheme, the data storage device for storing the list of non-redundant data values ​​is main memory. Main memory can be the processor's main memory, particularly the processor of a single-board computer such as a Raspberry Pi.

[0114] In another aspect, the present invention relates to multiple lists of non-redundant data values. The data value lists include multiple concept lists and a list of no-concepts. Optionally, the non-redundant data value lists may also include a global list. The non-redundant lists contain data values ​​and object IDs of data objects, each containing one or more of those data values. Each concept list represents a semantic concept, and the non-redundant lists selectively include data values ​​assigned to the semantic concepts within that concept list. Each data value in the concept list is assigned the object IDs of all data objects containing that data value, and the contained data values ​​represent the semantic concepts within that concept list. The no-concept lists selectively include data values ​​not assigned semantic concepts. Each data value in the no-concept lists is assigned the object IDs of all data objects containing that data value, and the data values ​​contained in those data objects are not assigned semantic concepts.

[0115] In another aspect, the present invention relates to the use of a list of non-redundant data values ​​for searching and / or analyzing the data values ​​contained in such lists. The list of non-redundant data values ​​is particularly suitable for use by electronic data processing systems. Typically, the search and / or analysis is performed by the same DVS system that created the list. However, the search may also be performed by another system (e.g., another DVS system) rather than the system that generated the list.

[0116] In another aspect, the present invention relates to a computer system configured to search for data in an electronic database, the computer system being configured to use a list of non-redundant data values ​​to search for and / or analyze the data values ​​contained in such lists during the search.

[0117] Furthermore, the present invention relates to a computer system for storing data. The computer system includes at least one processor, a data memory, and a data processing search system (DVS system). The DVS system is designed to manage and search data stored in the data memory and is designed to cause at least one processor to execute a method comprising:

[0118] The Data Processing Search System (DVS system) receives raw data or raw data access addresses, and the raw data has different structures;

[0119] - Multiple different parsers parse the raw data to determine data objects, each with one or more data values ​​and an object ID for each data object, wherein at least some data values ​​are assigned semantic concepts;

[0120] - The DVS system automatically imports the parsing results;

[0121] The DVS system automatically stores all parsing results in a data storage device as a list of non-redundant data values. These non-redundant lists include:

[0122] • One or more concept lists, each representing a semantic concept. The list without redundancy selectively includes the imported data values ​​assigned to the semantic concepts in the concept list during parsing. Each data value in the concept list is assigned the object ID of all data objects containing that data value. The data values ​​contained in the list represent the semantic concepts in the concept list.

[0123] • No-concept list: The no-concept list selectively includes imported data values ​​that were not assigned semantic concepts during parsing. Each data value in the no-concept list is assigned the object ID of all data objects that contain that data value. The data values ​​contained in the data objects were not assigned semantic concepts during parsing.

[0124] - The DVS system provides a non-redundant list to answer search queries and / or perform data analysis.

[0125] According to embodiments of the present invention, at least one processor includes an arithmetic logic unit (ALU) designed to perform set operations on sets of object IDs assigned data values ​​to two or more non-redundant lists. These set operations specifically include intersection, union, difference, or symmetric difference operations. In particular, the set operations can be performed by the ALU, thereby comparing two object IDs within a single ALU cycle (comparison operation).

[0126] In this article, "processor" refers to a programmable computing unit (often significantly miniaturized and often idle), i.e., a machine or electronic circuit that controls other machines or circuits to drive algorithms (processes) based on given commands, typically involving data processing. For example, a processor can be designed as a main processor, a central processing unit, or (more generally) a central processing unit (CPU) for use in computers or computer-like devices that execute instructions. Processors can also be designed as microcontrollers in embedded systems (such as home appliances, ticket vending machines, and smartphones).

[0127] According to an embodiment of the present invention, a subunit of the processor directly executes at least one step of the above method.

[0128] In particular, according to embodiments of the present invention, at least one processor's Arithmetic Logic Unit (ALU) can directly perform set operations on object IDs. The ALU can combine two binary values ​​of the same bit length (n), i.e., an n-bit ALU. Typical values ​​for n are 8, 16, 32, and 64. According to embodiments of the present invention, all object IDs have a fixed length, particularly the same length, preferably chosen such that each object ID fits perfectly into the ALU's working register, allowing for comparison within the ALU during set operations. Specifically, to compare two sets of object IDs, for example, to calculate intersection, union, difference, or symmetric difference, all object IDs in one set can be compared with all object IDs in the other set.

[0129] In this document, "raw data" refers to data in any electronic form that has not yet been made available for analysis by the analyzer of a DVS system. Raw data particularly includes data obtained directly during observation, measurement, or data collection but not yet processed. In the context of this invention, "raw data" also includes data derived from the initially collected data, but not performed by the analyzer used to import the data into the DVS system. Raw data should thus refer to data that is extracted and stored typically entirely independently of the presence or involvement of the DVS system.

[0130] In this article, "access address" refers to information that enables a data processing system to read at least the data available at those access addresses. Access addresses can be, for example, URLs of files available on a network, file addresses based on a local file system, database names and tables containing specific datasets, or similar information.

[0131] This document, "Data Processing Search System (DVS System)," refers to a software- and / or hardware-based system for storing, managing, and processing electronic data. According to embodiments of the present invention, the DVS system is designed to efficiently, consistently, and permanently store large amounts of data. According to embodiments of the present invention, the DVS system may include multiple components that can be designed as modules: an import component for receiving and parsing raw data and for storing the parsed data in a non-redundant list. Here, the import component can use existing lists and automatically create new lists as needed. The DVS system may also include a search analysis component for searching and / or analyzing the list. Optionally, the DVS system may include a GUI that allows users to input raw data to be parsed and / or concept / data value pairs (key-value pairs) and / or search terms and / or analysis commands to be imported. The DVS system has read and write access permissions to the data storage it manages. According to embodiments of the present invention, the DVS system includes a data storage device and lists stored therein. Optionally, the DVS system may also include a document storage device for storing at least some raw data.

[0132] In this document, "data object" or "dataset" refers to a set of content-related (belonging to a certain object) data values, such as product number, product name, and production date. A dataset corresponds to the logical structure of the data values, which is specified when storing the data of a data object or is only identified when parsing the data. For example, if the parsed raw data is an image, then the individual objects depicted in the image can be considered "data objects" in the sense of this invention. However, it is also possible that the entire pixel information matrix of the image constitutes the data object "image." In this case, the digital image is a data object, and the image identifier can be used as an object ID.

[0133] The data objects contained in the original data include both datasets that are explicitly identified as objects in the original data (e.g., rows of an Excel spreadsheet with row-by-row data) and data objects that are dynamically identified and extracted during the parsing process. The latter refers to data objects that are "implicitly" contained in the original data.

[0134] In this article, "data value" refers to the smallest evaluable unit of a dataset. The smallest evaluable unit depends on the application context in which the data value should be identified and extracted, and optionally, on the semantic meaning of that data value in the original data, such as the parser. For example, an image analysis parser might be designed to recognize patterns in digital images at the digital level. In this case, the data value would be a binary value linked to pixel location data. Alternatively, the parser might identify patterns based on the intensity value of a single pixel. Thus, the data value could be the intensity value at a particular pixel location. Depending on the type of original data, the data value could be a sequence of alphanumeric characters (e.g., words, peptide sequences, nucleotide sequences), numbers (e.g., weight or size information), or binary objects (images, audio, or video files).

[0135] In this paper, "parser" refers to a computer program or module responsible for decomposing and transforming input into a format more suitable for further processing. Parsers are particularly useful for identifying the meaning (semantics, semantic concepts) of data values ​​explicitly or implicitly contained in the input and forwarding the identified semantic concepts linked to those data values ​​for further processing. Typically, a parser outputs the processed raw data in the expected form, containing additional semantic information. For example, a parser used to import data into a DVS system can output the parsed data values ​​and their semantic concepts as key-value pairs, which can then be stored in an existing list or used as the basis for automatically creating new lists of concepts. However, a parser can also be a pure syntax parser, processing the data structure of the raw data according to a specific decomposition and / or extraction scheme to extract one or more data values ​​from the raw data.

[0136] The “list of data values ​​without redundancy” in this article refers to a list of data values ​​where each data value is contained only once.

[0137] In this document, "data storage" refers to a storage medium, a storage area on a storage medium, or a combination of multiple storage media or storage areas used for storing data. If a data storage device comprises multiple storage media or storage areas, they can be interconnected to form a logical data storage device. Here, the storage media or storage areas can be interconnected, for example, via a network or via a computer system bus. For example, a data storage device managed by a DVS system can be a data storage device with exclusive access to its data by the DVS system.

[0138] In this article, the "concept list" refers to a sorted list of non-redundant data values. Each concept list represents a specific semantic concept and contains only the data values ​​assigned to that semantic concept during import. The object IDs assigned to specific data values ​​or list elements in the concept list include only the object IDs of the data objects that contain that data value or list element and were assigned that semantic concept during import.

[0139] In this article, the "conceptless list" refers to a sorted list of non-redundant data values ​​that represents data values ​​without semantic concepts and includes data values ​​for which no semantic concepts were assigned during import. The object IDs assigned to specific data values ​​or list elements in the conceptless list only include the object IDs of data objects containing that data value or list element that were not assigned a semantic concept during import.

[0140] In this article, "global list" refers to a sorted list of non-redundant data values, where each data value in the global list is assigned one or more pointers, each pointer pointing to an element in one of the concept lists or an element in the no-concept list, which contains the same data value as that data value in the global list.

[0141] In this article, "semantic concept" refers to the meaning that can be realized in different specific embodiments. For example, a semantic concept can represent a class of objects or processes containing multiple elements. Depending on the type of the original data and / or the parser used, a large number of different concepts can be assigned to the parsed and imported data values. In the context of medical data, a semantic concept can be a diagnosis, such as including different specific embodiments (data values) such as "diabetes," "Parkinson's disease," or "skin cancer." Another semantic concept can be "symptoms," including embodiments or data values ​​such as "fever," "chills," and "toothache."

[0142] The term "Importing Parsing Results into the DVS System" in this article refers to the process by which the DVS system receives parsing results from one or more external parsers. If one or more parsers are built-in components of the DVS system, then importing involves transferring the parsing results from these parsers to the DVS system module responsible for storing the results in a non-redundant list format.

[0143] In this article, "computer system" refers to a microcontroller-based or distributed data processing system, especially a digital data processing system. Data processing systems can include, for example, standalone computer systems or computer networks, particularly cloud systems. Computer systems can also be designed as mobile data processing systems, such as laptops, tablets, or portable telecommunications devices like smartphones.

[0144] In this document, "system" refers to a combination of one or more components capable of processing data. For this purpose, system components exchange data and / or control commands. For example, a system may include a computer with a DVS system. Optionally, a system may include further components, such as one or more client computer systems (sending search queries and / or analysis requests to the DVS system) and / or source systems (both containing some raw data transformed by the DVS system and the parser in a list of non-redundant data values). Attached Figure Description

[0145] The embodiments of the present invention will now be described with reference to the accompanying drawings. In the drawings:

[0146] Figure 1 The system block diagram of the DVS system is shown;

[0147] Figure 2 A flowchart of the data storage method is shown;

[0148] Figure 3 It shows various data structures of raw data with different structures and contents;

[0149] Figure 4 This shows a selection dialog box for some lists generated by the DVS system;

[0150] Figure 5 A distributed computer system with DVS system clients is shown;

[0151] Figure 6 A screenshot of the GUI of the DVS system with regular expression search results is shown.

[0152] Figure 7 Screenshots of the DVS system's GUI with different search options are shown;

[0153] Figure 8 The screenshot shows the GUI of the DVS system, which summarizes the contents of the concept list. Detailed Implementation

[0154] Figure 1 A block diagram of a system 100 having a DVS system 102 according to an embodiment of the present invention is shown.

[0155] For example, Figure 1The system 100 shown can execute Figure 2 The flowcharts depict the data storage and retrieval methods. Therefore, the following description of System 100 and System 100 will refer to these two figures. Figure 2 The method.

[0156] For example, this system and method can be used to integrate large amounts of heterogeneous raw data 112, making it searchable and scalable in an efficient manner. Raw data may include XML files, JSON files, text files in different formats (e.g., Microsoft Word, OpenOffice, Adobe Acrobat PDF files, TXT files, etc.), different tables from one or more different relational databases, media data, or hierarchically organized data, such as object trees. In some implementations, raw data can also be input through the user interface of the DVS system. Typically, raw data comes from multiple independent data sources, such as from various organizations, particularly companies, institutions, and agencies. In some raw data, data objects, their data values, and optional semantic meanings can be more or less explicitly demonstrated, such as in database tables, Excel files, and other relatively complex data with corresponding fields. In other raw data (e.g., image data), data objects and their data values ​​are implicit rather than explicit. This means that data objects and their data values ​​are only identified and extracted during the parsing process.

[0157] To parse different raw data, several different syntax and / or semantic parsers 118, 120, 122, 124, 126, 128, and 130 are provided. For example, a parser may be a component of the integration module 110 of the DVS system 102. A parser may also be an external component initiated and controlled by the DVS system. In order to parse the raw data, the parser 118, 120, 122, 124, 126, 128, and 130 used must at least read and access the corresponding raw data. During parsing, the parser preferably exchanges information with the DVS system or other components of the DVS system, for example, to determine whether an object ID is truly unique or already occupied by an object ID used as a reference in a list of data values ​​when dynamically creating an object ID for a data object.

[0158] In the first step 202, the DVS system 102 and / or a parser controlled by it receive the raw data 112 or at least an address that accesses the raw data. The access address at least allows reading access to the raw data. For example, the raw data may be generated or acquired in one or more source systems connected via a network and transmitted to the computer system 100 via a network (e.g., the Internet). In some embodiments, the DVS system stores a copy of the raw data, for example, for later reprocessing by additional and / or improved versions of an existing parser. However, the raw data is not directly processed to respond to search queries.

[0159] Parsing and importing different raw data can be done in parallel, or they can be processed sequentially, even with long intervals, depending on the availability of the raw data and / or the parser used for parsing.

[0160] Step 204 involves multiple parsers 118, 120, 122, 124, 126, 128, and 130 processing the raw data to extract data objects, their object IDs, and one or more data values ​​contained in each data object. For example, different parsers can be assigned to Excel files, text files, image files, different XML files, and database tables. The object ID can be a data object identifier already specified by the raw data and taken over as the object ID by the DVS system. Alternatively, object IDs can be generated only during parsing. For example, multiple datasets can be stored row-by-row in an Excel spreadsheet. The parser for this Excel spreadsheet can be configured to interpret each row of the Excel file as a data object, using a combination of the filename and row number containing the row of the dataset / data object. If the extracted data values ​​have been explicitly assigned a corresponding semantic concept in the raw data (e.g., through appropriate field or column identifiers), the relevant parser assigns the semantic concept embodied by the extracted data values ​​to the extracted data values. Because the parser uses certain field identifiers as semantic concepts, if a concept list does not yet exist in the data storage, the DVS system will automatically generate the corresponding concept list and learn the new semantic concepts automatically. Data values ​​extracted from the raw data during parsing are stored in the concept list according to the semantic concepts assigned to their field identifiers. However, the parser may also only identify data objects and data values ​​not explicitly included in the raw data during complex parsing processes. The parser may also fail to identify which semantic concept the extracted data value belongs to. After the parsing step, the result may be a (potentially redundant) list of data values, with each element linked to an object ID and optionally assigned a semantic concept.

[0161] Step 206: The DVS system imports the results of the parsing process through the integration module 110, and stores the results in the form of a list of non-redundant data values ​​114, 115, and 116 in the data storage or data storage area 104 managed by the DVS system.

[0162] These lists include one or more concept lists 116, typically a list of concepts with and without, and an optional global list 115. Preferably, the data values ​​within the lists are sorted according to a specific sorting relationship, i.e., for example, alphabetical order. See below for further details. Figure 3 This section details the functionality and structure of the list. This storage format addresses, to some extent, the issue of object structure—that is, what data values ​​or semantic concepts exist within a specific data object. This allows for powerful data compression, and the list can be expanded at any time with additional original data and / or additional data values ​​and / or concepts provided by the parser, without requiring structural changes.

[0163] In step 210, the DVS system provides non-redundant lists 114, 115, and 116 to respond to search queries and / or analyze the data values ​​contained in the lists. For example, the DVS system creating the lists may include a search query receiving and processing module 106 and / or an analysis module. For example, module 106 may be configured to receive search queries from one or more client computers via a network interface or directly from local working users via a GUI. As a supplement or alternative, module 106 may also include a large number of complex analysis functions based on set operations on multiple concept lists, and optionally also on set operations on non-concept lists and global lists.

[0164] In other implementations, the DVS system that creates lists 114, 115, and 116 may also transfer the originals or copies of lists 114, 115, and 116 to the search analysis system. The search analysis system may be another instance of the DVS system, or other hardware- and / or software-based data processing systems with interfaces for receiving and processing search queries and / or analysis commands. Optionally, such other search analysis systems may also include a parser and an interface for importing parsed results, but are not necessarily required to include such components.

[0165] Figure 3 The diagram shows data structures with raw data, which have different structures and different contents.

[0166] For example, data structures 302, 304, and 306 are JSON format product data sheets for a manufacturer's motors (shown as tab-delimited text files in this diagram due to space constraints). The parsing process can be accomplished as follows: each of the three JSON files 302, 304, and 306 is interpreted into its own data object, each with an object ID. Each data object contains several key-value fields, such as specific data values ​​for the semantic concept of "power," specific data values ​​for the semantic concept of "torque," etc.

[0167] The data structure shown below is an Excel spreadsheet containing specifications for different paint properties from paint distributors. Each row (308, 310, 312, 313) contains exactly one data object (dataset). During parsing, each identified data object can be assigned an object ID, such as the row number combined with an identifier from the Excel spreadsheet.

[0168] Some of the raw data may be in text format, such as text files 314, 316, and 318. For example, this text can be broken down into individual words using a pure parser (each word serving as a data value). Alternatively, the parser could be a tokenizer, breaking down natural language text into words used as data values ​​(excluding some possible stop words). In this case, the words / data values ​​are initially unassigned semantically upon import; the words and their corresponding source object IDs are stored in a list of unassigned terms, with each word appearing only once in this list.

[0169] Another portion of the raw data can be provided, for example, in the form of business registration books 320, 322, and 324. These can contain a mixture of key-value fields and free text.

[0170] The raw data comes from different sources, but sometimes the content overlaps ("Gelb AG"), and sometimes the data values ​​are ambiguous ("Silver"). The embodiments of this invention, however, allow for efficient integration and processing of all this data while resolving semantic ambiguity.

[0171] Figure 4 The DVS system is shown to be based on Figure 3 The diagram shows some selection boxes for data structures generated from the original data. The data values ​​displayed in the non-redundant lists 114, 115, and 116 are also just one selection; the lists are usually much longer.

[0172] The lists 114, 115, and 116 of all data values ​​generated and managed by the DVS system are non-redundant lists, meaning they contain each data value only once. Preferably, the data values ​​are also sorted so that, based on the order of the search terms and the searched list data values, a sequential search in the list can be terminated if further searching in the list is unlikely to result in a match.

[0173] For example, the parser specifically identifies the following semantic concepts when processing raw data: color, manufacturer, paint number, cash on delivery, relationship type, and metal type. These semantic concepts are represented by concept lists 402, 404, 406, 408, 410, and 412, respectively.

[0174] The no-concept list 115 contains data values ​​for which no semantic meaning was assigned during parsing. For example, if the parser for message text 314 is a plain text word segmenter and cannot recognize the meaning of words, then the word "no concept" determined by the parser is stored in the no-concept list in a non-redundant form. Therefore, if text 314 mentions "Gelb AG," and the parser used does not recognize the semantic meaning of the word, then this data value will be stored in the no-concept list, referencing data object 314. The same data value "Gelb AG" may also be contained in other raw data, and thus, when parsing these other raw data, a semantic meaning can be assigned to this data value. For example, the data value "Gelb AG" is also contained in the business register 320 under the field identifier "Firma (Company)." For example, the business registration parser and / or DVS system can be configured to treat the semantic concepts "Firma (company)" and "Hersteller (manufacturer)" as synonyms of the same semantic concept, so that the data value "Gelb AG" in the concept list 404 of the concept "Hersteller" is stored as an object ID linked to the data object 302. If the data value already exists in the list, only the set of object IDs assigned to that data value is added accordingly.

[0175] No concept list 115 and Figure 3 Other lists shown demonstrate that the list data values ​​for different data objects are sorted, and the sorting of data values ​​in the lists is independent of the source of the data values ​​for a particular data object. Thus, from a structural perspective, the original structure of the data objects in the original data is fully resolved when the lists are generated, because data values ​​assigned to data objects can only be provided in the form of object IDs and can be reconstructed.

[0176] Alternatively, in some implementations, the location data of the data value within each data object can also be stored together with the object ID in lists 115 and 116, thereby reconstructing the location of the data value within the data object in the original data.

[0177] The list of non-redundant data values ​​preferably also includes a global list 114. This list does not contain object IDs for associating data values ​​with one or more semantic concepts and corresponding concept lists. For example, the data value "silver" in the global list is assigned the concept lists "color," "surname," and "metal type," while only the data value "Stolze" is assigned the concept list "surname." For instance, the global list can be used to dynamically display different semantic concepts containing a specific search term as a search term is entered and / or to restrict data values ​​evaluated during database searches or data analysis to those representing specific semantic concepts.

[0178] Preferably, the list also includes one or more concept lists 410 representing data object relationships. Here, the data value is a concatenation of relationship type (e.g., "Verbaubar in") and object ID (e.g., identifiers for motor types MF-3000,...,MF-6000). This relationship specifies which motor type a component (specified in the data object from which the relationship is extracted during parsing) can be fitted into. A search query with the search value "Verbaubar_in_MF_3100" can quickly return the object IDs of all data objects representing motor components that can be fitted into MF_3100 type motors. This provides a highly efficient search opportunity for all possible object relationship types, and is also usable by data processing devices with limited main memory and / or computing power.

[0179] Figure 5 A distributed computer system 500 with DVS system clients 506 and 508 is shown, which sends search queries and / or analysis requests to interface 106 of DVS system 102 via a network and receives responses, such as a list of objects whose data values ​​match the submitted search criteria.

[0180] The raw data is also provided by multiple source systems 502 and 504 via a network (e.g., the Internet).

[0181] However, Figure 5 The system architecture 500 shown is just one of many possible architectures. There are many alternative architectures, such as storing the original data as a copy on the same computer system 100 as the DVS system and / or using one or more local client requests instantiated on the same computer system 100, as a supplement or replacement to client systems 506, 508 connected via a network. Parsers 118, 120, 122, 124, 126, 128, and 130 can be components of the DVS system or external software programs controlled by the DVS system.

[0182] For example, parsers 118, 120, 122, 124, 126, 128, and 130 can first generate a key-value list (KV list) from the extracted information, which still contains data values ​​in redundant form, such as color = green, weight = 100 kg, music genre = classical, book = science, etc. As a supplement or alternative, the KV list can be received through the GUI of the data processing system. In this case, the parsing process only involves accepting the specified key-value pairs and connecting them to the object ID of the data object from which the key-value pairs are retrieved. For example, the GUI could allow a doctor to make a diagnosis based on multiple symptom inputs. A "case (Fall)" can be input as a continuous dataset (data object), and the key-value pairs "symptom = high blood sugar level," "symptom = low insulin concentration," "symptom = dizziness," and "diagnosis = diabetes" are stored in a "symptom" or "diagnosis" concept list, and associated with the same case by searching for the object ID of that "case."

[0183] However, in addition to reconstructing data values ​​for specific cases / patients, comprehensive statistical analysis can be performed on a list of data values ​​containing symptoms and diagnoses of hundreds of thousands of patients.

[0184] This allows for the acquisition of new medical outcomes through efficient correlation analysis of non-redundant lists, such as by finding the intersection of object IDs from data values ​​of multiple concept lists.

[0185] All key-value pairs 503 generated by parsers 118, 120, 122, 124, 126, 128, and 130 are normalized, preferably automatically, and always processed in the same way before being transferred to lists of non-redundant data values ​​114, 115, and 116. No changes are required to the list structure or any indexes (in traditional DBMSs, both must typically be adjusted when integrating more data or new datasets with previously unknown structures).

[0186] Therefore, the list of data values ​​as a whole represents a multidimensional model in which the dataset does not exist as a separate unit structurally separated from other datasets. The affiliation of a data value with a specific data object is calculated only by searching the list without redundancy and / or performing set operations on the object IDs of the elements in these lists.

[0187] For example, a search query could contain search criteria for phone book entries such as: "Vorname= / [CK].*ristin / ", "Straβe=" / .*[Aa]llee.* / " and "Hausnummer= / 14 / ".

[0188] The search query consists of two regular expressions as search criteria that allow for "fuzzy matching" and a data value "14" that must be exactly satisfied. The DVS system is configured to identify a list of concepts (in this case, names, streets, and house numbers) to which the search criteria are applied, and then perform a fuzzy search or search for exactly the same value in the data values ​​of each concept list based on the criteria type. This search is not only extremely fast, but also has the advantage of not ignoring phone book entries (data objects) that match the search criteria if there are typos or spelling errors in the name, provided that the typo is still included in the regular expression.

[0189] Figure 6 A screenshot of the GUI 600 of a DVS system with regular expression search results according to one embodiment is shown.

[0190] In the example shown in this figure, an 8.9GB raw dataset is parsed and transformed into a 4.42GB list without redundancy. The raw dataset consists of a set of comma-separated files, each containing a large number of telephone data sets, with each column containing different semantic concepts such as last name, first name, street, house number, zip code, city, area code, and telephone number. As described in the previous implementation, this data is transformed into a global list and a list of concepts, each containing only non-redundant data values: last name, first name, zip code, city, street, area code, and telephone number. Furthermore, for some datasets (data objects), relationships with other data objects (data sets) are defined; in this example, the relationship is "mother" and "father."

[0191] The GUI allows users to specify different search queries, which can include search criteria for one or more lists, and can be used for different search patterns (identity or regular expression matching). The search pattern can be selected individually for each concept index.

[0192] Figure 6In the example shown, a regular expression is entered into search field 602 to search for the regular expression within the non-redundant concept list "Last Name". The first step determines all the data values ​​of the regular expressions that "match" the "Last Name" concept list and assigns the object IDs to these matching data values ​​in that concept list. This first step takes only 0.7 seconds because it doesn't search the original 89GB of data, but only the sorted list of last names with 3.2 million distinct data values. The second step uses all the object IDs determined in the first step to output data objects (datasets) representing those whose last names "match". In the example shown in this figure, outputting these datasets takes approximately 11 seconds. For such runtimes, it must be considered that they were measured on a 2012 MacBook Air with 8GB of RAM, which, in addition to the DVS system used for data searching, also ran a web server and various other programs, storing the non-redundant list via an external SSD connected to the laptop via USB. Searching using regular expressions on an older laptop from several years ago, rather than on a dedicated database server, would take less than a second.

[0193] The advantage of using a list of non-redundant data values ​​is that it allows for highly efficient identity searches and highly efficient searches using regular expressions. This specific data organization in the form of a list of non-redundant data values ​​also supports a wide range of other search queries, some of which can be found in [link to relevant documentation]. Figure 7 .

[0194] Figure 7 Screenshots of the DVS system GUI 700 with different search options are shown. On one hand, the user can choose to perform a "global search," referred to here as a "general search." When using a general search, the user enters search terms for the identity search or enters search terms in the form of regular expressions. The search is then performed in a global list containing all imported data values. For example, the global list might contain the search term "Berlin," which references several concept lists assigned to the global list, "Last Name," "First Name," "City," and "Street," appearing 1327, 1112234, 89412, or 81 times respectively. In contrast, the data value "Berlin" does not appear in the "House Number" and "Zibo Code" concept lists. The 81 occurrences of the name "Berlin" might suggest an error in the original data, such as a missing or incorrect delimiter, which could cause the field assignments to be unrecognizable during parsing.

[0195] As a supplement to or alternative to global search, users can also define and execute arbitrary combined searches. Combined searches allow users to enter search criteria for each semantic concept that needs to be considered in the search (e.g., name, abbreviation, and telephone number or street address). Preferably, each time a new letter is entered in any search criterion, a search is triggered for all (all or part of) the entered search criteria.

[0196] Depending on the expression type, the execution time for several regular expressions on several concept lists ranges from a few seconds to a fraction of a second. It is noteworthy that no indexes need to be created, nor is the list structure (or index) adjusted based on the number and type of semantic concepts to be searched. In many traditional DBMSs, in addition to the tabular relational data (which is approximately equivalent to 8.9 GB of raw data), a large number of search indexes must be created. Thus, the efficient memory consumption of traditional relational DBMSs supporting equal-volume search queries is several times higher than the 4.42 GB consumed by the DVS system according to an embodiment of the present invention for various possible combinations of search queries on the global list and / or concept list and / or non-concept list.

[0197] For example, a 550GB original dataset containing 1.4 billion taxi trips (including departure and arrival times, passenger numbers, and payment amounts) can be transferred to a set of non-redundant lists totaling 396GB. The original datasets (data objects) reference one taxi region as the origin and another as the destination. Given this list organization, despite the large database size, determining the total number of taxi trips originating or destined for a specific region takes only 0.01 seconds.

[0198] Figure 8 The screenshot shows the DVS system's GUI 800, which summarizes the contents of the concept list. The "Feldname (field name)" column contains the name of the concept list managed by the DVS system and used for searching. This name is exactly the same as the semantic concept represented by that concept list. The "(Field Index Entry)" column displays the number of distinct data values ​​in each concept list. The "Telefon" concept list contains 6,941,134 distinct phone numbers (without area codes). The "Name" concept list contains 3,210,129 distinct last names. The "Vorname" concept list contains 673,264 distinct first names. The "Vorkommen" column displays the collective (distinct, i.e., non-redundant) object IDs assigned to all data values ​​in each list. Thus, the original data contains 34,795,673 data objects (datasets) with data values ​​in the "Telefon" field, which collectively contain and extract a total of 6,941,134 distinct phone numbers.

Claims

1. A method for storing data in a data storage device (104), the method comprising: The data processing search system (102) receives (202) raw data (112) or raw data access address, wherein the raw data has a different structure; Multiple different parsers parse (204) the original data to obtain data objects and object IDs for each data object, wherein each data object has one or more data values, at least one of the one or more data values ​​being assigned a semantic concept, wherein the multiple different parsers include at least one of a syntax parser and a semantic parser, wherein the syntax parser is a word segmenter that decomposes natural language text in the original data into words, and the semantic parser identifies the imported data values ​​and assigns semantic concepts to the data values ​​based on data analysis, wherein the data analysis is image analysis, audio signal analysis, statistical analysis, classification, machine learning, and / or pattern recognition. The data processing and search system automatically imports the (206) parsing results; The data processing and search system automatically stores (208) all parsing results in the data storage device in the form of a list of non-redundant data values ​​(114, 115, 116), wherein the list of non-redundant data values ​​includes: One or more concept lists (116), wherein each concept list represents a semantic concept, and a list of non-redundant data values ​​selectively includes imported data values ​​assigned to the semantic concepts in the concept lists during parsing, wherein each data value in the concept list is assigned an object ID to all data objects containing the data value, wherein the included data values ​​are representations of the semantic concepts in the concept lists; A list of data values ​​without semantic meaning (115), wherein the list of data values ​​without semantic meaning selectively includes imported data values ​​that were not assigned semantic meaning during parsing, wherein each data value in the list of data values ​​without semantic meaning is assigned the object ID of all data objects containing the data value, wherein the data values ​​contained in the data objects were not assigned semantic meaning during parsing; and The data processing search system provides (210) the list of non-redundant data values ​​(114, 115, 116) to respond to search queries and / or perform data analysis, thereby performing search queries and / or data analysis without accessing the original data.

2. The method of claim 1, wherein, At least one piece of raw data exists or is received in the form of multiple data structures, wherein the multiple data structures are a mixture of two or more of the following data structures: -XML file; - JSON file; -Text file; -CSV file; - Database tables; -Object tree; -Media files; - Data received via the GUI; - Streaming data.

3. The method of claim 2, wherein, The media files are video files, audio files, and / or image files.

4. The method of any one of claims 1 to 3, wherein, The list of non-redundant data values ​​also includes: A global list (114), wherein the global list is a list of all imported data values ​​without redundancy, wherein each data value in the global list is assigned one or more pointers, wherein each pointer points to an element in one of the concept lists or an element in the no-concept list, the element containing the same data value as the data value; The data processing search system is configured to analyze or query the global list at least, wherein the global list is used to identify and / or process data values ​​representing different semantic concepts in different data objects for concept processing.

5. The method according to any one of claims 1 to 3, wherein, At least one data value is extracted from a field in a data structure containing the original data, wherein the field is specified by the data structure, and wherein the field includes one or more concept-related fields and / or one or more concept-independent fields, wherein the concept-related fields are fields assigned field identifiers, wherein the field identifiers represent semantic concepts; and / or, the concept-independent fields are fields not assigned semantic meaning; and / or At least one data value is imported by the semantic parser; The data processing and search system stores the data values ​​extracted from the concept-related fields in a concept list of semantic concepts representing the field identifiers of the concept-related fields; and / or If the parser used does not recognize the semantic concept of the data value, the data processing search system will store the data value extracted from the concept-independent field only in the concept-free list (115).

6. The method according to any one of claims 1 to 3, further comprising: After importing and storing data values, an additional parser is provided, the additional parser being configured to identify and import data values ​​that are given at least one new semantic concept, wherein the new semantic concept is a concept not represented by any concept list contained in the current data storage; The additional parser processes the original data, wherein it extracts one or more new data values ​​from the original data that give rise to at least one new concept; The data processing search system compares at least one new semantic concept identified by the additional parser with the concept list in the data storage, and for each new semantic concept, the additional parser extracts at least one data value, automatically generating and storing the new concept list. The data values ​​extracted from the original data by the additional parser are automatically stored in the at least one new concept list, which represents the new concept list to which the additional parser assigns new semantic concepts to the data values.

7. The method according to claim 4, further comprising: Receive a search query that identifies data objects that satisfy one or more concept-related search conditions and / or one or more concept-independent search conditions, wherein the concept-related search conditions are search conditions that assign semantic concept identifiers, and the concept-independent search conditions are search conditions that do not assign semantic concepts. Search the list of no-concepts for one or more data values ​​that satisfy each received concept-independent search condition; and / or For each received concept-related search condition, selectively search the list of concepts representing the semantic concepts of the search condition for one or more data values ​​that satisfy the concept-related search condition; Returns the object ID or a subset of these object IDs that the data processing search system determines in response to a search query to search the global list and / or the at least one concept list.

8. The method of any one of claims 1 to 3, wherein, Using the aforementioned list of non-redundant data values ​​(114, 115, 116) for query searching and / or data analysis includes: Perform set operations on a set of object IDs assigned to data values ​​in two or more lists of non-redundant data values, wherein the set operations include intersection, union, difference, or symmetric difference operations.

9. The method according to claim 4, wherein At least one data object is related to other data objects; In the process of parsing the original data and / or later performing data processing operations on the original data and / or the data values ​​of the stored list of non-redundant data values, the data processing search system extracts the object relationships between these data objects and other data objects. In this context, each extraction relationship between a data object (i.e., the first data object) and another data object (i.e., the second data object) is extracted as a combination of the relationship type and the object ID of the second data object; and Each extracted combination is stored as one of the data values ​​in the list of no concepts and / or the at least one concept list, wherein each extracted combination in the list of no concepts and / or the at least one concept list is assigned an object ID of a first data object that has a specified relationship in the combination.

10. The method of claim 9, further comprising: Receive a search query or analysis command that identifies all data objects related to the search data object, wherein the search query or analysis command includes a search object ID; The global list (114) is searched to determine all data values ​​composed of relation types and object IDs that are the same as the search object ID; Identify one or more concept lists (116) related to the data values ​​determined in the global list; Search one or more identified lists of concepts to determine data values ​​composed of a relation type and an object ID that is the same as the search object ID; Returns the object IDs of one or more concept lists that assign the determined data value, i.e., the object IDs of the data objects that are related to the search object.

11. The method according to any one of claims 1 to 3 and 10, further comprising: The data processing search system provides a mapping table, wherein the mapping table assigns one or more identifiers to each data value in the no-concept list and each data value in the concept list, wherein the identifiers are values ​​whose length and / or type depend on the processor architecture of the computer system used for search and / or analysis; The data processing search system generates the list of no concepts and a confused copy of each concept list, wherein the data values ​​in the confused copies are all replaced with the identifiers assigned to the data values ​​in the mapping table; The data processing search system uses the obfuscated copy to perform the search query and / or analysis.

12. The method according to any one of claims 1 to 3 and 10, further comprising: The data processing search system encrypts the concept list and the no-concept list, or the data processing search system encrypts the obfuscated concept list and the obfuscated no-concept list; In response to receiving a search query or data analysis command, identify an encrypted list that needs to be processed to handle the search query or data analysis command; The data processing search system verifies the sender of the search query, wherein the data processing search system checks whether the sender is authorized to access the list content for each identified list. The data processing search system selectively generates decrypted copies of the identified list, which have been verified to grant the sender access to it. Perform search queries or data analysis on the decrypted copy of the list; After performing the search query or data analysis, the data processing search system deletes the decrypted copy of the list, wherein the data processing search system uses different encryption keys to encrypt each list.

13. The method according to claim 4, wherein At least one list of concepts represents genomic markers of multiple individuals of a species or subspecies, wherein at least one other list of concepts represents metabolic or phenotypic characteristics of these individuals, wherein the data processing search system performs data analysis through list set operations, wherein the data analysis involves correlation analysis between the genomic markers and the metabolic or phenotypic characteristics; or Wherein, at least one data value in the global list and / or the at least one concept list specifies an "affiliation" relationship with other objects, including the object ID of the other objects, wherein at least one other concept list represents the metabolic or phenotypic characteristics of these individuals, wherein the data processing search system performs a query search, wherein the search criteria involve a combination of the relationship type "affiliation" and the object ID, wherein the object ID of the search criteria represents a machine or vehicle or multiple components of the machine or vehicle; or Wherein, at least one data value in the global list and / or the at least one concept list is a natural language word, wherein the data processing search system performs data analysis through list set operations, wherein the data analysis involves word correlation analysis to create a predictive model for the natural language.

14. The method of any one of claims 1 to 3 and 10, wherein, Sort the data values ​​in the list of non-redundant data values ​​(114, 115, 116).

15. A volatile or non-volatile storage medium having stored thereon computer readable instructions, wherein, The instructions are designed to cause the processor to execute a method for storing data in a data memory according to any one of the preceding claims.

16. A computer system for storing data, comprising: At least one processor (108); Data storage (104); A data processing search system (102), wherein the data processing search system is designed to manage and search data stored in the data memory (104), and is designed to cause the at least one processor to execute a method comprising: The data processing search system (102) receives (202) raw data (112) or raw data access address, wherein the raw data has a different structure; Multiple different parsers parse (204) the original data to obtain data objects and object IDs for each data object, wherein each data object has one or more data values, at least one of the one or more data values ​​being assigned a semantic concept, wherein the multiple different parsers include at least one of a syntax parser and a semantic parser, wherein the syntax parser is a word segmenter that decomposes natural language text in the original data into words, and the semantic parser identifies the imported data values ​​and assigns semantic concepts to the data values ​​based on data analysis, wherein the data analysis is image analysis, audio signal analysis, statistical analysis, classification, machine learning, and / or pattern recognition. The data processing and search system automatically imports the (206) parsing results; The data processing and search system automatically stores (208) all parsing results in the data storage device in the form of a list of non-redundant data values ​​(114, 115, 116), wherein the list of non-redundant data values ​​includes: One or more concept lists (116), wherein each concept list represents a semantic concept, and a list of non-redundant data values ​​selectively includes imported data values ​​assigned to the semantic concepts in the concept lists during parsing, wherein each data value in the concept list is assigned an object ID to all data objects containing the data value, wherein the included data values ​​are representations of the semantic concepts in the concept lists; A list of data values ​​without semantic meaning (115), wherein the list of data values ​​without semantic meaning selectively includes imported data values ​​that were not assigned semantic meaning during parsing, wherein each data value in the list of data values ​​without semantic meaning is assigned the object ID of all data objects containing the data value, wherein the data values ​​contained in the data objects were not assigned semantic meaning during parsing; and The data processing search system provides (210) a list of non-redundant data values ​​(114, 115, 116) to respond to search queries and / or perform data analysis.

17. The computer system of claim 16, wherein, The at least one processor includes an arithmetic logic unit, which is designed to perform set operations on a set of object IDs assigned to data values ​​in two or more lists of non-redundant data values. The set operations include intersection, union, difference, or symmetric difference operations.

Images