List-based data storage for data retrieval.

Abstract

The present invention relates to a method for storing data (114, 115, 116) in a data storage (104), the method comprising: receiving (202) raw data (112) or an access address of the raw data by a data processing and retrieval system (DVS system) (102); - parsing (204) the raw data with a plurality of different parsers to identify data objects each having one or more data values ​​and a respective object ID of the data object, at least some of the data values ​​having respective semantic concepts assigned to them; - a step (206) of automatically importing the parsed results of the DVS system; - a step (208) of automatically storing all parsed results in the form of a redundancy-free data value list (114, 115, 116) in a data storage by the DVS system, The redundant list is: one or more concept lists, each of the concept lists represents a semantic concept; the non-redundant list selectively includes imported data values ​​that were assigned the semantic concepts of the concept list during said parsing; A non-concept list, the no-concept list selectively including imported data values ​​that were not assigned a semantic concept during the parsing; providing the no-redundancy list by the DVS system for responding to search queries and / or for performing data analysis.

Description

Technical Field

[0001] The present invention relates to a method and system for processing and storing data for retrieving data.

Background Art

[0002] In the prior art, various database management systems (DBMSs: Datenbankmanagementsysteme) are known for storing, managing, and efficiently processing data. The essential problem of a DBMS is to efficiently, consistently, and permanently store a large amount of data and enable a user or application program to use a required subset in various forms of expression according to needs. The basis for structuring the data and its relationships in a database managed by a conventional DBMS is a database model defined by the manufacturer of the DBMS. Depending on the database model, the database schema (Datenbankschema) needs to be adapted to specific structuring options. Known database models currently in use include hierarchical, network, relational (table-organized), object-oriented, document-oriented, and their mixed types. Furthermore, classically, DBMSs optimized to efficiently respond to a large number of small-scale queries (OLTP) or long-term evaluations (OLAP) are distinguished.

[0003] In a conventional DBMS, when selecting a DBMS or defining the internal structure of a database managed by the DBMS (such as the size, number, internal references of database tables, selection of column types for creating indexes, etc.), it is necessary to consider both the content (amount of information) of the data managed by the database developer and the types of queries (Anfragen) that the DBMS will have to process.

[0004] The problem is that these two aspects can change over time and are often unknown or completely unknown at the time of database setup. In practice, other or additional aspects of the data stored in the database often turn out to be relevant over time, requiring the formulation of new queries that could not have been considered during the initial database creation. For example, if a database contains medical data, and after the database is set up, new medical knowledge emerges that certain symptom combinations can predict a particular diagnosis, the database data may still contain symptoms, but they may be distributed and / or indexed across different tables, thus making queries inefficient, i.e., consuming a lot of working memory, CPU capacity, and time. However, subsequent structural adjustments to a database once defined are often very time-consuming, error-prone, and no longer possible due to the numerous dependencies between the data in the database and the client system.

[0005] Furthermore, as time progresses, additional data sources must be integrated into the database, but the internal structure of these sources often does not conform to the data model selected during the database setup. For example, if a database with a relational data model is selected and hierarchical data needs to be integrated, even if the additional data is conceptually compatible, it is often not possible to store the additional data in that manner within the database. Moreover, even if it is possible to store the additional data in an inherently unsuitable data model, common queries and semantic integration between the existing data and the additional data will be impossible unless the data structure within the database is fundamentally revised.

[0006] Thus, existing DBMSs often suffer from structural inflexibility, low scalability, and poor performance, especially when handling complex queries on large amounts of data objects with many different attributes (keys) and corresponding values. Subsequently, if data with further different structures must be stored in the database, it becomes impossible to query and parse existing and newly added data together in a content-meaning, high-performance, and resource-efficient manner. This is a particular problem in the context of the "Internet of Things," but not only in that context. This is because various objects and sensors collect data that is highly heterogeneous in content and structure, and its composition and properties often change over time. [Overview of the project]

[0007] The objective of this invention is to propose an improved method and system for storing data in an efficiently searchable manner.

[0008] The fundamental problems of the present invention are solved by the features of each independent claim. Embodiments of the present invention are shown in the dependent claims. The embodiments shown below can be freely combined with each other, provided that they are not mutually exclusive.

[0009] In one embodiment, the present invention relates to a data storage or a method for storing data in a data storage (Datenspeicher). This method includes the following: - A data processing and retrieval system - A DVS system that receives raw data or an access address for raw data, wherein the raw data has a different structure; - A step of parsing raw data using multiple different parsers, wherein each data object has one or more data values ​​and respective object IDs, and each data value is assigned a semantic concept; - Steps to automatically import parsing results from the DVS system; - A step in which the DVS system automatically saves all parsing results in the form of a non-redundant data value list within the data storage, The aforementioned list without redundancy is: A concept list is a list of one or more concepts, each of which represents one semantic concept, and the non-redundant list selectively includes imported data values ​​that have been assigned the semantic concepts of that concept list during parsing, and each data value in the concept list is assigned all object IDs of the data object containing that data value, and the included data values ​​are representations of the semantic concepts of that concept list. A step including a conceptless list, which selectively includes imported data values ​​to which no semantic concepts were assigned during parsing, and each data value in the conceptless list is assigned all object IDs of the data object containing that data value, and to which no semantic concepts could be assigned during parsing for the data values ​​contained in that data object, - A method comprising the step of providing a non-redundant list by a DVS system in order to respond to a search query and / or to perform data analysis.

[0010] This transfers highly heterogeneous raw data in terms of structure and content into a common structure (intersection, join, difference, or symmetric difference set of object IDs of elements in multiple lists) that can process large amounts of data quickly and efficiently, without requiring fundamental changes to the data structure in data storage for this purpose, and is advantageous in that it can be expanded at any time in terms of content (and even more advantageously by adding automatically generated concept lists) and the queries and analysis processes it supports. Because lists are non-redundant, each data value is included only once in a list. If it appears in multiple data objects in the raw data, this data value appearing once in the list is stored linked to these multiple object IDs. In this way, even very large datasets can be processed quickly and efficiently on computer systems with limited working memory and / or CPU capacity. In particular, when the raw data contains a particular data value multiple times (for example, in the case of text representing a specific combination of a finite set of existing words, or in the case of protein representing a specific sequence of a finite set of amino acids), this form of data representation leads to enormous data compression effects. For example, an object ID can represent a dataset (such as a protein sequence ID, a URL of a raw data file, natural language text or a document, or a row in an Excel file), and a data value is a value that occurs within that dataset (such as a word, a bit string, or a number).

[0011] Assigning semantic concepts to data values ​​is typically part of the parsing process.

[0012] According to embodiments of the present invention, the DVS system thus stores imported data values ​​in a non-redundant data value list according to the semantic concepts assigned to them (which may be determined during parsing or otherwise predetermined), thereby storing and distributing data values ​​in the list independently of their original affiliation with data objects.

[0013] Thus, embodiments of the present invention have the advantage of being highly flexible with regard to the integration of large amounts of structurally and contentically heterogeneous data, and of having very high performance in data analysis and retrieval processes, because the assignment of data values ​​to data objects is structurally resolved when imported into data storage: the assignment of data values ​​to objects exists only in the object ID and does not affect the structure of the data managed by the DVS system: regardless of whether the raw data is provided as hierarchical, tabular, or XML files, whether the data values ​​are already largely predefined in the raw data (e.g., by key-value fields), or whether they are dynamically obtained by one of the parsers in the analysis process (e.g., pattern recognition of image or audio data), the data values ​​are always stored in a non-redundant list linked to the object ID containing the data value. This storage takes into account the semantic concept that the data value represents and selectively stores the data value in a concept list that represents that concept. If the data value already exists in this list, only the set of object IDs is appropriately supplemented. If the semantic concept behind the data value could not be identified when parsing the raw data, the data value is stored in a "no concept" list, and if the data value already exists there, only the corresponding entry is extended by the object ID.

[0014] Thus, the import process can be understood as a process that resolves the entire data object structure by saving the sum of all data values ​​extracted from all data objects in the raw data to a list of data values ​​without redundancy, based on the semantic concepts assigned to them.

[0015] For example, different data sources such as JSON files, XML files, database tables, media files, or user input can be provided to the DVS system via a user interface, such as a graphical user interface. This provision can be done, for example, by having the DVS system receive raw data (original or copy) from local data storage or other sources over a network and first store it in its own document store. This has the advantage that the original is still available and, in some cases, can be used for the purpose of displaying search or analysis results, or for parsing and analysis that is only supplemented later. However, it is also possible that the raw data is not stored as a copy in the DVS system's document store, and the DVS system only receives (permanent or temporary) read access to the raw data so that it can process and parse it. In this case, it is sufficient for the DVS system to receive the access address of the raw data.

[0016] Embodiments of the present invention have the advantage that data objects described by a very large number of properties (values) associated with a wide variety of semantic concepts (keys) can be queried for any combination of diverse key values ​​in very short query times, without following defaults that depend on the original structure of the data object (for example, as in search queries in relational, index-based DBMSs relating to table and index structures in a database). In index-based systems, indexes for all possible combinations of keys must be available. Therefore, in conventional index-based DBMSs, the set of indexes increases in proportion to the factorial of the key! In particular, for many different object types with many different semantic concepts, the number of indexes required increases with the factorial of the key! to every conceivable combination of search criteria associated with the key. On the other hand, according to embodiments of the present invention, there is no need to generate and use indexes (searchable data structures generated in addition to data values, e.g., in the sense of a B-tree). According to embodiments, a non-redundant concept list corresponds to each semantic concept, and each data value of this concept appears only once, in contrast to tables in a data record base (datensatzbasierten) of a relational DBMS, for example. Therefore, according to embodiments of the present invention, searches and / or analyses can be performed without creating an index structure that conforms to the expected search queries. Rather, searches can be performed directly in a list without redundancy, preferably using only the list of concepts corresponding to the semantic concepts referenced by the search criteria (and optionally, a list without concepts and / or a global list as well).

[0017] According to embodiments of the present invention, the data storage is volatile or non-volatile data storage. The data storage can be, for example, a hard disk drive (HDD), a mass storage device - solid state drive (SSD), or working memory (memory storage and list management).

[0018] According to embodiments of the present invention, at least a portion of the raw data is in the form of a plurality of data structures, or is received in the form of a plurality of data structures. In particular, the plurality of data structures may consist of a mixture of two or more of the following data structures: - XML ​​file - JSON file - Text file - CSV file - Database Table - Object tree - Media files (especially video, audio, and image files) - Data entered via GUI - Streaming data

[0019] Streaming data is data that is continuously generated by a source system and transmitted in queues and / or packets (based on the FiFo principle). Typically, this data is processed in "real time" by a streaming framework. The streaming framework receives the stream of this data, processes the information in memory, and then writes it to data storage (e.g., a hard disk drive (HDD) or mass storage device - solid state drive (SSD)). Depending on the embodiment, a DVS system can also be designed for real-time processing of streaming raw data. Streaming data may include, for example, log data from an ERP system, e-commerce events (views, orders, baskets), tracking events from a mobile app, geolocation from a web application, or usage data for a specific product.

[0020] However, this list should not be understood as exhaustive.

[0021] According to an embodiment, at least some (typically most) of the non-redundant lists contain data values that were included in two or more of the different data structures containing raw data. Therefore, the question of in which data structure a data value occurs in the raw data does not affect the structure of the lists managed by the DVS system. This is advantageous in the sense of decoupling the system from the structure of files and databases containing raw data. The raw data can be data of various contents and structures. This can be advantageous because the use of the DVS system enables the integration of large amounts of heterogeneous data from many independent sources into a common data structure (non-redundant data value list). Thereby, not only can the amount of data be highly compressed, but at least if the raw data from two or more different sources has at least one type of data value representing the same semantic concept and this is recognized during the parsing process, semantic integration is also possible. In this case, the data values of this concept are stored linked to the references to the respective data objects within the same concept list.

[0022] Also, it is particularly advantageous that it is not necessary to fully semantically integrate data objects from different sources. For example, if the raw data is a protein sequence or a gene sequence and during the parsing process of these data, it is recognized that a specific peptide sequence or genomic marker representing the concept of "predictor of diagnosis X" also exists in the raw data of other data sources, it does not matter if the parser used cannot recognize and extract all the semantic concepts that may exist in the raw data of a certain data source. Thus, a complete "understanding" or "data model" is not required to integrate data from different heterogeneous sources. However, when parsing the raw data of different data sources, if at least one or some data values of a specific semantic concept are found, they are classified into the corresponding list and can thus be integrated.

[0023] Even when the semantic concept is unclear, it is possible to integrate heterogeneous data based on correlation: For example, if a certain genomic marker is frequently described in the literature together with a certain diagnosis name, on the one hand, the genomic marker is included, and on the other hand, the common part or cut set (Schnittmengen) of the object IDs of the data objects including the diagnosis becomes very large (larger than the statistically expected value considering the occurrence frequency of individual data values). When the biological function behind this marker and thus the related semantic concept are unclear, the marker is stored as a data value in the list without concepts, so that during the processing of search queries, data analysis, such as correlation analysis, etc., regardless of whether the genomic marker is provided in the form of an XML file, a text file, or any other format, it can be considered.

[0024] Embodiments of the present invention thus also have the advantage of being very flexible and extensible: The structure of the data objects of the raw data, as well as the type and number of attributes of the data objects, do not affect the structure of the data managed by the DMS system (these are all stored in the form of non-redundant lists anyway), so new raw data can be imported and integrated into a new structure at any time. When the structure of the raw data has been unclear until now, process the raw data, and if possible, assign the data values contained therein to semantic concepts, and according to the existence and type of the assigned semantic concepts, it is only necessary to develop a corresponding parser to store the data values in one of the list without concepts or the concept list together with the object ID of the related data object or at least the object ID.

[0025] According to one embodiment, the method includes performing search queries and / or analysis using a DVS system or another DMS system.

[0026] According to the embodiment, search queries and / or data analysis are performed without accessing raw data, in particular.

[0027] According to the embodiment, the DVS system is configured to perform analysis and / or retrieval solely on non-redundant lists, or on data derived from these lists by the DVS system, and the analysis and / or retrieval is not performed on raw data, the data objects contained therein, or data records (Datensaetzen) representing these data objects.

[0028] This has the advantage that searching and analysis does not depend on the structure of the raw data, and even terabytes of raw data can be processed quickly on a standard computer and in some cases on a single-board computer such as a Raspberry Pi, and can be compressed and processed to a size that can be searched with complex queries. Compression relies on each data value occurring only once in each concept list and in the no-concept list. For example, if a data value such as "silver" is assigned to multiple different semantic concepts (surname, color, metal type), it may appear in two or more concept lists. However, in each list, the value "silver" appears only once. If the data value "silver" is found in further data objects representing the concept "metal," the object IDs of these data objects are selectively assigned to the list element "silver" in the list representing the concept "metal type." When the parser parses certain raw data and cannot recognize whether the data value "silver" represents a color, metal, surname, or other semantic concept that has not been previously represented as a list, this data value "silver" is recognized as being in the no-concept list. Therefore, even if the data value "silver" appears thousands of times in the raw data, it may appear only once or a few times in the list on which the DVS system operates. Object IDs are preferably stored in numerical form, which consumes little memory and allows for fast set operations. Thus, complex search queries, for example, all data objects including the product type "pipe material" and "silver" as a "metal type," can be responded to by calculating the intersection of object IDs from the sets of object IDs assigned to the data value "pipe material" in the concept list "product type" and the data value "silver" in the concept list "metal type." For this reason, the structure of data objects contained in the raw data becomes unnecessary for data analysis.

[0029] According to one embodiment, the data storage managed by the DVS system does not contain data objects (sometimes referred to as data records).

[0030] In other embodiments, the DVS system additionally manages a document store where all or at least part of the data structure, including raw data, is stored. However, here again, this raw data is used for purposes other than processing search queries or data analysis. For example, it may be advantageous to store a copy of the raw data in the DVS system's document store (Dokumentenspeicher) in order to run a new parsing process on the raw data after additional and / or better parsers become available, in order to detect and extract further data values ​​and / or semantic concepts assigned to them that could not be detected or parsed by the previously used parser. This reduces network traffic because the raw data stored on the network-connected system does not need to be retransmitted when a new or better parser becomes available.

[0031] According to embodiments of the present invention, the non-redundant list further includes a global list. The global list is a non-redundant list of all imported data values, and each data value in the global list is assigned one or more pointers, each of which points to an element of a concept list or a non-concept list containing the same data value as that data value. The DVS system is configured to perform analysis or query searches (angefragte Suche) on at least the global list, and the global list is used to recognize and / or process concept-specific data values ​​representing different semantic concepts in different data objects.

[0032] Preferably, data values ​​in a global list are not assigned object IDs. For example, a global list contains the data value "silver" only once. This data value in the global list is not directly assigned an object ID, but is assigned a pointer to one or more concept lists representing the semantic concepts to which the data value "silver" is assigned in at least one data object. In this example, the data value "silver" in the global list would thus be assigned the concept lists "Last Name," "Metal Type," and "Color." If the data value "silver" is also stored in a no-concept list, the no-concept list would also be assigned to the "silver" data value.

[0033] Using a global list has the advantage of allowing direct determination of each data value whether it represents a different semantic concept depending on the context, and if so, how many and which concepts it represents. Preferably, object IDs are directly assigned only to data values ​​in individual concept lists or lists without concepts, so that data objects containing the data value "silver" according to one concept can be easily selected and distinguished from data objects containing the word "silver" according to other concepts.

[0034] According to one embodiment, the DVS system is designed to provide a GUI for entering search terms (eines Suchbegriffs). While entering search terms, the DVS system searches a global list to determine whether the entered term corresponds to a data value in the global list. The DVS system suggests to the user one or more identified data values ​​that are at least partially identical or similar to the already entered search terms. For example, the DVS system may present identified data values ​​to the user in a context menu or autocomplete list to provide options. When the user selects one of the suggested data values, the DVS system identifies the number and types of semantic concepts (and corresponding concept lists) that this data value in the global list refers to. The thus identified semantic concepts are also displayed on the GUI, allowing the user to select one or more of these semantic concepts via the GUI. For example, when the user enters the search term "silver," they learn that several concepts (surname, metal type, color) are known for this search term. Users can select these concepts using GUI elements such as checkboxes, dropdown lists, and radio buttons, thereby limiting search queries to specific semantic concepts. For example, selecting the "metal type" concept restricts the search to data objects containing the data value "silver" as the metal identifier. The object IDs of these data objects are all stored in the "metal type" concept list and assigned to the "silver" list element.

[0035] According to embodiments of the present invention, at least a portion of the data values ​​are extracted from fields of a data structure containing raw data. The fields are defined by the data structure and include one or more concept-related fields and / or one or more concept-unknown fields (konzeptagnostische Felder).

[0036] A field can be, for example, a cell in an Excel file, an element in a database table, or a specific XML element in an XML object tree. It corresponds to a predefined container within a data structure used to store one or more data values. A concept-related field (Ein konzeptbezogenes Feld) is a field to which a field identifier is assigned, and the field identifier represents a semantic concept. A concept-unknown field is a field that has no semantic meaning assignment. Typically, at least some of the data values ​​of the raw data are stored in concept-related fields, and / or some of the other data values ​​of the raw data are stored in concept-unknown fields. For example, if an Excel table is parsed and neither its columns nor rows contain identifiers for the semantic concepts of the data contained in the table, those fields may be considered concept-unknown fields. However, if field identifiers such as "Last Name," "First Name," "Date of Birth," "Date Manufactured," and "Weight" exist, then they are concept-related fields.

[0037] Parsing and importing data values ​​stored in concept-related fields is advantageous because it allows new semantic concepts to be automatically "learned" and integrated into the DVS system. The use of several parser types can also achieve immeasurable learning benefits: parsers can be used by defining a parser that interprets the data values ​​of cells in the first row of a table as a semantic concept that interprets the data values ​​below that cell in the same column. New semantic concepts and new data values ​​can be imported from an Excel table following a scheme where the semantic concepts for each column are included in the first row (title). If the DVS system recognizes that the semantic concept "Last Name" is previously unknown (i.e., no concept list exists for "Last Name") when importing data values ​​for the "Last Name" column, the DVS system will automatically create such a concept list, and all data values ​​for this column will be stored in this list without redundancy.

[0038] According to an embodiment of the present invention, the DVS system stores the data values ​​extracted from concept-related fields in the data values ​​of a concept list representing the semantic concepts of the field identifiers of these concept-related fields.

[0039] According to one embodiment, if the parser used does not recognize the semantic concept of the data value, the DVS system stores the data value extracted from the concept-ignorant field only in a concept-ignorant list.

[0040] According to embodiments of the present invention, at least a portion of the data values ​​are imported by a semantic parser that recognizes not only the imported data values ​​but also the semantic concepts assigned to them based on data analysis, and the data analysis is in particular image analysis, audio signal analysis, statistical analysis, classification methods, machine learning methods, and / or pattern recognition methods. According to one embodiment, the parsers used include a parser that classifies audio data and converts it into text, breaking it down into words to be used as data values. Additionally or alternatively, the parser can evaluate the audio data to recognize a music genre, and the entire audio data can be treated as data values ​​to which the automatically identified music genre (e.g., rock, techno, classical) is assigned as a semantic concept.

[0041] Therefore, even if there is no field identifier indicating the semantic concept to which a data value is assigned, a semantic concept may be recognized during the parsing process. For example, the raw data may be a digital image, and the parser may be image analysis software capable of recognizing a given object (person, animal, plant, etc.). An object ID is assigned to the image or image segment in which a specific object is recognized, and only the image (or the pixel data of the corresponding image segment) or the name of the recognized object ("human," "dog," "cat," etc.) is stored as a parsed "data value" in the corresponding concept list ("human" or "dog," etc.). Therefore, whether a data value can be recognized and extracted from the raw data, and whether a semantic concept can be assigned to this data value, may depend on the field identifier and / or the parser used.

[0042] If the non-redundant data value list does not yet contain a concept list for a dynamically detected new semantic concept (a new music style, a new object type for image data), the DVS system preferably automatically creates a new concept list for that semantic concept.

[0043] According to embodiments of the present invention, this method further: - A step of further providing a parser designed to recognize and import data values ​​that have been imported and saved, and which have been assigned to at least one new semantic concept, wherein the new semantic concept is a concept not represented by any of the concept lists currently contained in the data storage; - A step of processing raw data with a further parser, the step of extracting one or more new data values ​​from the raw data assigned to at least one new concept; - A step of comparing at least one new semantic concept recognized by a further parser with a list of concepts in the data memory of the DVS system, and automatically generating and storing a new concept list for each of the at least one new semantic concepts from which the further parser has extracted at least one data value; - The process includes the step of automatically saving the data values ​​extracted from the raw data by a further parser to at least one new concept list representing the new semantic concepts assigned to those data values ​​by the further parser. - Preferably, the no-concepts list and / or global list are also updated so that, in the no-concepts list, the object IDs assigned to data values ​​of data objects that contain data values ​​to which the new parser was first able to assign a new concept are removed. These object IDs are then assigned to the data values ​​in the new concepts list, respectively. Object IDs of data objects that contain data values ​​but to which the new parser was not able to assign a concept remain in the no-concepts list for the sake of the data values.

[0044] In this way, the same raw data can then be evaluated by a new, improved parser, and the knowledge extracted in that process can be seamlessly integrated into the existing list structure and made searchable.

[0045] According to the embodiment, the method further: - A step of receiving a search query to identify data objects that satisfy one or more concept-related search criteria and / or one or more concept-unknown search criteria, wherein the concept-related search criteria are search criteria to which a semantic concept identifier is assigned, and the concept-unknown search criteria are search criteria to which the semantic concept is not assigned, - A step of searching a list of no concepts for one or more data values ​​that satisfy each of the received concept-unknown search criteria, wherein the search can be, for example, a search for data values ​​that are identical or similar with respect to the search term; for similarity searches, for example, a regular expression can be used; and / or, - For each of the received concept-related search criteria, the step of selectively searching for one or more data values ​​that satisfy the concept-related search criteria from the concept list representing the semantic concept of that search criterion; - The process includes the step of returning, in response to a search query by the DVS system, object IDs assigned to data values ​​identified during a search of the global list and / or at least one concept list, or a subset of those object IDs.

[0046] Based on the compression of a massive amount of data values ​​and their division into different lists according to semantic concepts, even vast amounts of raw data can be processed very quickly, even on low-performance computers, after being converted into data values ​​contained in a list structure.

[0047] According to a preferred embodiment of the present invention, each data value in the list is sorted, that is, ordered according to an ordering relationship (for example, alphabetically for data values ​​consisting of alphanumeric strings). This speeds up the search of the list for a particular data value because, based on the ordering relationship, the search can be stopped if it is determined that the data value cannot be included in the list after passing through the data values ​​in the list.

[0048] According to one embodiment, using a non-redundant list to perform query lookup and / or data analysis includes the step of performing a set operation or set operation (Mengenoperation) on a set of object IDs assigned to two or more data values ​​in the non-redundant list, wherein the set operation includes, in particular, an operation on an intersection set, a join set, a difference set, or a symmetric difference set.

[0049] For example, multiple concept lists and / or no-concept lists can each recognize one or more data values ​​("match values") that are identical or at most similar to the query search terms. By applying an integration set formation operation (Vereinigungsmengenbildungsoperation) to the set of object IDs assigned to each of these match values ​​in the concept lists, the integrated whole of data objects containing all the match values ​​in that one concept list is identified. For different concept lists, by applying an intersection set formation operation to the set of object IDs obtained in the integration set formation step, the intersection set of data objects containing at least one match value for each of the concept lists referenced by the query search criteria is identified. This intersection set can be returned as a result of the search query and / or used for further analysis. According to the embodiment, the data values ​​of the no-concept list and / or one or more concept lists are each stored linked to one or more time information (Zeitangaben), each time information including the transaction time of the initial generation of the data value or the time of import of the data value by the DVS system. This method further includes considering time information by a DVS system in order to respond to search queries and / or perform data analysis.

[0050] For example, the time "from March 13, 1978" can be assigned to the data value "Frankfurt" in the concept list "Firmensitz" (Company Location) in relation to the object ID of commercial register extract number 238 for the company "Gelb AG," and time information can be obtained from there. Furthermore, the time "from July 16, 2012" can be assigned to the data value "Berlin" in the concept list "Company Location" in relation to the object ID of commercial register extract number 46474, and this time information, to which the same company "Gelb AG" is assigned, can be obtained from there. Therefore, by querying all the "Registered Office" data values ​​for "Gelb AG," it is possible to reconstruct the time series of company location changes because the data object from which the data value is extracted and its object ID are linked to this time information. The time series of company location changes can also be reconstructed by presenting all the "Company Location" data values ​​for "Gelb AG," because the data object from which the data value is extracted and its object ID are linked to this time information.

[0051] According to embodiments of the present invention, at least some data objects have relationships with other data objects. These relationships can be, for example, "is a component of ~", "is included in ~", "belongs to a class", "is activated", "is prohibited", or "is available in ~". The relationships are already explicitly or implicitly included in the raw data and can be extracted during the parsing process. However, it is also possible that the DVS system extracts the relationships only during data processing operations performed later on the raw data and / or data values ​​in a list of already stored non-redundant lists.

[0052] Each extracted relationship between one data object ("first data object") and another data object ("second data object") is extracted in the form of a combination (specifically, a konkatenation) of a relationship type (i.e., a relationship identifier such as "is_buildable_in") and the object ID of the second data object. Each extracted combination is stored as one of the data values ​​in the no-concept list and / or at least one concept list, and in each no-concept list and / or at least one concept list of the extracted combination, the object ID is assigned to the object ID of the first data object in which the relationship specified in that combination exists.

[0053] For example, such object relationship extraction and storage can be used to form an "open" search query that extracts the context of a particular object, including the set of objects that reference this search object and the nature of those relationships.

[0054] According to one embodiment, the object ID of a data object identified during parsing is also treated as a data value and is stored, for example, in the concept list "Object ID" (and optionally in the global list). For example, the object ID of a data object can be extracted as a data value and stored in the concept list "Object Identifier". This object identifier concept list can be searched using a specific object ID (search object ID) to find a data value identical to the search object ID within this list. Subsequently, in the object identifier concept list, the IDs of all data objects that can be assigned to this identified data value are identified. These include not only data objects identical to the search object, but also the object ID of the search object, and therefore all data objects that have a relationship with the search object.

[0055] In addition to the "identical" match search described above, it is also possible to search the global list using a regular expression that includes the object ID of the search object. For example, the global list can contain multiple data values ​​that form relationships such as "ist_verbaubar_in_Motor_XL-3000", "ist_leistungsequivalent_zu_Motor_XL-3000", and "wird_aktiviert_von_p53_Gen", where the component "Motor_XL-3000" or "p53_Gen" can represent an object ID. Based on a regular expression search that specifies only the object ID as the search object ID and opens up the relationship type identifier, all data values ​​that include the search object ID as components of the data value, i.e., all relationship types that refer to the search object, can first be identified in the global list. The data values ​​in the global list thus identified refer to the corresponding context lists, for example, the context lists for the semantic concepts "ist_verbaubar_in" and "ist_leistungsequivalent_zu". The concept list "ist_verbaubar_in" (available in ~) may contain the data value "ist_verbaubar_in_Motor_XL-3000" (available in Motor_XL-3000) to which the first set of object IDs is assigned. The concept list "ist_leistungsequivalent_zu" (equivalent output to ~) may have the data value "ist_leistungsequivalent_zu_Motor_XL-3000" (equivalent output to Motor_XL-3000) to which the second set of object IDs is assigned. Thus, the combined set of the first and second sets of object IDs results in the search object "Motor_XL_3000" containing the entirety of either the "verbaubar in" or "leistungsequivalent zu" data objects.

[0056] According to embodiments of the present invention, the method is: - A step of receiving a search query or analysis command to identify all data objects that have a relationship with a search data object, wherein the query or command includes a search object ID, - A step of searching a global list to identify all data values ​​that consist of a combination of a relationship type and an object ID identical to the search object ID, - The steps of identifying one or more concept lists that refer to the data value identified in the global list; - A step of searching one or more identified concept lists to identify a data value consisting of a combination of the relation type and an object ID identical to the search object ID, - The step of returning, in one or more concept lists, the object IDs assigned to the identified data values, i.e., the IDs of the data objects that have a relationship with the search object.

[0057] Therefore, it is possible to execute very complex search queries and similarity searches (fuzzy searches using regular expressions) in a highly flexible, scalable, and high-performance way, without having to create a B-tree database index and adapt it when integrating new data object types or search queries as needed. Thus, for each search object, it is possible to very quickly find out what relationship types actually exist in relation to the search object, and how many, and what other objects, reference or connect to the search object through one or more of these relationship types.

[0058] This is particularly advantageous in the context of applying "Industry 4.0" or when ordering and / or configuring highly complex components (such as aircraft turbines and engines). Complex components or system components can consist of thousands of components, and those components can be composed of subcomponents. When a database of an automobile or aircraft manufacturer contains a large number of components, this type of relational query allows for very quick determination of which data objects have a relationship such as, for example, "ist-Bestandteil-von-Motor-XM-300049 BT".

[0059] According to embodiments of the present invention, the method is - A step of providing a mapping table using a DVS system, wherein the mapping table assigns one or more identifiers to each data value in the no-concept list and each data value in the concept list. - A step of generating a list without concepts and an obfuscated copy of each concept list using a DVS system, wherein in the obfuscated copy, each data value is replaced with an identifier assigned to the data value in the mapping table. - The steps include: using an obfuscated list to perform search queries and / or analysis using a DVS system.

[0060] This has the advantage of improving data security. For example, copies of the mapping table and the obfuscated list can be provided on different computer systems and / or stored and used in separate IT security environments. If an unauthorized third party accesses the obfuscated list, they cannot recognize the contents of the list without the mapping table. Furthermore, the mapping table itself does not contain object IDs that would allow the assignment of data values ​​to objects to be reconstructed, so access to it is harmless. Therefore, for an unauthorized third party to reconstruct the original list, they would need to infiltrate two separate computer systems or two separate security systems. In this way, a particularly secure data management and retrieval method is provided.

[0061] According to the embodiment, the identifier is a value whose length and / or type depends on the processor architecture of the computer system used for retrieval and / or analysis, i.e., it is selected according to these factors. In particular, the length and / or type of the identifier is selected so that it can be processed very quickly by the processor architecture. For example, the length of the identifier can correspond to the processing width of the processor architecture (e.g., 32 bits for a 32-bit architecture, 64 bits for a 64-bit architecture). If the processor architecture can process numbers particularly efficiently, the identifier can consist of numbers. If the processor architecture can process other types of values ​​(e.g., symbols) particularly efficiently, the identifier can consist of symbols.

[0062] According to the embodiment, the search or analysis identifies a search value identifier based on a mapping table, and the search value identifier is an identifier assigned to a search value in the mapping table.

[0063] According to embodiments of the present invention, all identifiers have a fixed length, particularly the same length, and preferably, each identifier is selected so that it fits entirely within the work register of the ALU of at least one processor performing the method according to embodiments of the present invention. Thus, according to embodiments of the present invention, the identifiers are compared with each other and / or with a lookup value identifier in a single work cycle of the ALU.

[0064] This can be advantageous not only because it improves the protection of data content from unauthorized access by replacing data values ​​with one or more identifiers, but also because it can provide more efficient data processing during the process of handling search queries and data analysis.

[0065] According to embodiments of the present invention, this method further: - A step of encrypting the concept list and the no-concept list using a DVS system, or a step of encrypting the obfuscated concept list and the obfuscated no-concept list using a DVS system; - In response to receiving a search query or data analysis command, the steps include identifying the list from the encrypted list that needs to be processed in order to process the search query or data analysis command; - A step of authenticating the sender of a search query to a DVS system, wherein the DVS system individually checks, for each identified list, whether the sender has permission to access the contents of the list; - The DVS system selectively generates a decrypted copy of the list of which the sender has been authenticated as having access rights; - The step of performing a search query or data analysis on the decrypted list copy; - The process includes the step of deleting the list copy decrypted by the DVS system after the execution of a search query or data analysis.

[0066] This has the advantage that lists generated and managed by the DVS system are essentially unreadable because they are encrypted. The encryption keys required for decryption are preferably stored securely on an HSM or in a relatively secure environment. This means that an unauthorized third party who obtains a list or a copy of a list cannot read the data they contain. Only in response to a search query or analysis command are parts of the list that need to be read to answer the query temporarily decrypted. Thus, it is possible to provide a flexible, scalable, resource-efficient, and secure method for managing and retrieving data.

[0067] According to a preferred embodiment, the DVS system uses a different encryption key to encrypt each item in the list. This further enhances security.

[0068] According to the embodiment, at least one of the concept lists represents genomic markers for multiple individuals of a species or subspecies, and at least one other of the concept lists represents metabolic or phenotypic characteristics of these individuals. The DVS system performs data analysis using set operations on the lists. Data analysis includes correlation analysis of genomic markers on the one hand and metabolic or phenotypic traits on the other. Thus, a novel and highly efficient form for conducting so-called "genome-wide association studies (GWAs)" is provided.

[0069] According to another embodiment, at least a portion of the data values ​​in the global list and / or at least one concept list specify an "ist-Bestandteil-von" relationship with other objects, including the object ID of the other object; at least one other concept list represents the metabolic or phenotypic characteristics of these individuals; the DVS system performs a query search, the search criteria including a combination of the relationship type "ist-Bestandteil-von" and an object ID, where the object ID of the search criterion represents a machine or vehicle or one of the multiple components of that machine or vehicle.

[0070] According to other embodiments, at least some of the data values ​​in a global list and / or at least one conceptual list are words in natural language. The DVS system performs data analysis using set operations on the lists. The data analysis includes word correlation analysis to create a predictive model of this natural language.

[0071] In another aspect, the present invention relates to a volatile or non-volatile storage medium storing computer-readable instructions, wherein the instructions are designed to cause a processor to perform a method of storing data in a data memory as described in any one of the preceding claims.

[0072] According to one embodiment, the data memory in which the non-redundant data value list is stored is the main memory (Arbeitsspeicher). The main memory may be the main memory of a processor, particularly in a single-board computer, such as a Raspberry Pi computer.

[0073] In another embodiment, the present invention relates to a plurality of non-redundant data value lists. A data value list includes a plurality of concept lists and one non-concept list. Optionally, a non-redundant data value list also includes a global list. A non-redundant list includes a data value and the object IDs of one or more data objects that each contain this data value. Each of the concept lists represents a semantic concept, and a non-redundant list selectively includes data values ​​to which the semantic concepts of this concept list are assigned, with each data value in the concept list being assigned the object IDs of all the data objects that contain this data value, and the included data values ​​are representations of the semantic concepts of this concept list. A non-concept list selectively includes data values ​​to which no semantic concepts are assigned, with each data value in the non-concept list being assigned the object IDs of all the data objects that contain this data value, and the included data values ​​of these data objects are not assigned semantic concepts.

[0074] In a further embodiment, the present invention relates to using a non-redundant data value list to retrieve and / or analyze the data values ​​contained in the list. In particular, the use of a non-redundant data value list may be performed by an electronic data processing system. Typically, the retrieval and / or analysis is performed by the same DVS system that created the list. However, it is also possible that the retrieval is performed by a system different from the one that generated the list, for example, a different DVS system.

[0075] In another embodiment, the present invention relates to a computer system configured to retrieve data in an electronic data repository, wherein the computer system is configured to use a non-redundant data value list for retrieving and / or analyzing the data values ​​contained in the list during the retrieval.

[0076] In another embodiment, the present invention relates to a computer system for storing data. This computer system comprises at least one processor, a data memory, and a data processing and retrieval system—a DVS system. The DVS system is designed to manage and retrieve data stored in the data memory and has at least one processor, - A data processing and retrieval system - A step of causing a DVS system to receive raw data or an access address for raw data, wherein the raw data has a different structure, - A step of parsing raw data using multiple different parsers, each of which identifies a data object having one or more data values ​​and respective object IDs for the data objects, wherein at least some of the data values ​​are each assigned a semantic concept; - Steps to automatically import parsing results from the DVS system; - A step in which the DVS system automatically saves all parsing results in the form of a non-redundant data value list within the data storage, The list without redundancy is: A concept list is a list of one or more concepts, each of which represents one semantic concept. The non-redundant list selectively contains imported data values ​​that were assigned the semantic concepts of the concept list during parsing, and each data value in the concept list is assigned all the object IDs of the data object containing that data value. The included data values ​​are representations of the semantic concepts of the concept list. • A list without a concept, which selectively includes imported data values ​​to which no semantic concepts were assigned during parsing, and each data value in the list without a concept is assigned all object IDs of the data object containing that data value, and to which no semantic concepts could be assigned during parsing the data values ​​contained in this data object, and Steps including, - A step of providing a non-redundant list by the DVS system in order to respond to a search query and / or to perform data analysis, It is designed to perform methods that include this.

[0077] According to the embodiment, at least one processor includes an arithmetic logic unit - ALU - which is designed to perform set operations on a set of object IDs assigned to two or more data values ​​in a non-redundant list, the set operations including, in particular, intersection, join, difference, or symmetric difference operations. In particular, the set operations may be performed by the ALU such that the comparison of two object IDs is performed within one operating clock (comparison operation) of the ALU.

[0078] Here, "processor" is understood to mean a programmable arithmetic unit (usually very small and usually freely programmable), i.e., a machine or electronic circuit, that controls other machines or electrical circuits according to transferred instructions, driving algorithms (processes), usually involving data processing. A processor can be, for example, a main processor, a central processing unit, or (more generally) a central processing unit (CPU) for a computer or computer-like device that executes instructions. A processor can also be a microcontroller in an embedded system (such as a consumer electronics appliance, ticket machine, or smartphone).

[0079] According to one embodiment, at least one of the steps of the method is performed directly by a subunit of the processor.

[0080] In particular, according to embodiments of the present invention, set operations on object IDs can be performed directly by an arithmetic logic unit (ALU) of at least one processor. The ALU can concatenate two binary values ​​having the same number of digits (n). This is referred to as an n-bit ALU. Typical values ​​of n are 8, 16, 32, and 64. According to embodiments of the present invention, all object IDs have a fixed length, particularly identical lengths, and preferably each object ID is selected so that it fits completely into the ALU's work register and can be compared with each other in the course of set operations in the ALU. In particular, to compare two sets of object IDs, for example to compute an intersection set, a join set, a difference set, or a symmetric difference set, the ALU can verify identity by comparing all object IDs in one set with all object IDs in the other set.

[0081] As used herein, "raw data" means any data available in electronic form that has not been parsed by the parser of the DVS system. In particular, raw data includes data obtained directly during observation, measurement, or data collection that is available in its unprocessed state. However, "raw data" in the sense of this invention also includes data derived from primary collected data, the derivation of which has not been performed by the parser used to import data into the DVS system. Therefore, raw data should be understood to mean data whose acquisition and storage are typically generated and stored entirely independently of the presence or intervention of the DVS system.

[0082] An "access address (Zugriffsadressen)" is understood to be information that enables a data processing system to have at least read access to data that has become available under the access address. An access address can be, for example, a URL to a file available over a network, a local file system-based address of a file, or an indication of a database containing a specific data record and the name of a database table within it.

[0083] "Data Processing and Retrieval System - DVS System" is understood herein to mean a software and / or hardware-based system for storing, managing, and processing electronic data. According to embodiments of the present invention, a DVS system is designed to store large amounts of data efficiently, consistently, and persistently. According to embodiments, a DVS system may include several components that can be designed as modules: an import component for receiving and parsing raw data and storing the parsed data in non-redundant lists, where the import component can use existing lists and, if necessary, can automatically create new lists. The DVS system may further include a search and analysis component for searching and / or analyzing lists. Optionally, the DVS system may include a GUI on which a user can input raw data to be parsed and / or concept-data-value pairs (key-value pairs) and / or search terms and / or parsing commands to be imported. The DVS system has read and write access to data storage managed by the DVS system. According to embodiments, the DVS system includes data storage and lists stored therein. Optionally, the DVS system may further include a document store for storing at least a portion of the raw data.

[0084] A “data object” or “data record” is understood here as a group of data values ​​(belonging to an object) that have relevant content, such as part number and part name, and manufacturing date. A data record corresponds to the logical structure of data values ​​that is determined when the data of a data object is stored, or recognized only when the data is parsed. If the parsed raw data is, for example, an image, then each recognized object depicted in the image can be a “data object” in the sense of this invention. However, the entire matrix of pixel information of an image may constitute a data object “image”. In this case, the digital image is a data object, and the image identifier can be used as the object ID.

[0085] Therefore, the data objects included in the raw data include not only data records explicitly identified as individual objects in the raw data (for example, individual rows in an Excel table with row-level data records), but also data objects dynamically recognized and extracted during the parsing process. The latter refers to data objects that are "implicitly" included in the raw data.

[0086] A "data value" is understood as the smallest evaluable unit of a dataset. What is considered the smallest evaluable unit depends on the application context, e.g., the parser that recognizes and extracts the data values ​​in the raw data and optionally their semantic meaning. For example, an image analysis parser may be designed to recognize patterns in a digital image at the bit level. In this case, the data value is a binary value associated with pixel location data. In another example, a parser may recognize patterns based on the intensity values ​​of individual pixels. In this case, the data value may be the intensity value at a particular pixel location. Depending on the nature of the raw data, the data value may be an alphanumeric sequence (e.g., words, peptide sequences, nucleotide sequences), a numerical value (e.g., weight or size information), or a binary object (image, audio, or video file).

[0087] Here, "parser" is understood as a computer program or program module that takes input and breaks it down into a format suitable for further processing. In particular, a parser can recognize the meaning (semantics, semantic concepts) of data values ​​explicitly or implicitly contained in the input and pass on the recognized semantic concepts of extracted data values ​​linked to these data values ​​for further processing. Typically, a parser outputs the raw data it processes in a desired format, including additional semantic information. For example, a parser used to import data into a DVS system might output the parsed data values ​​and their semantic concepts in the form of key-value pairs, which can then be stored in an existing list or serve as the basis for automatically creating a new concept list. However, a parser can also be a purely syntactic parser that processes a data structure containing raw data according to a specific decomposition and / or extraction scheme and extracts one or more data values ​​from the raw data.

[0088] A "non-redundant" data value list is understood here as a data value list that contains each data value exactly once.

[0089] "Data storage" refers to a storage medium or storage area on a storage medium, or a combination of multiple storage mediums or storage areas, used to store data. When data storage includes multiple storage mediums or storage medium areas, these can be connected to each other to form logical data storage. In this case, the storage mediums or storage medium areas can be operably connected to each other, for example, via a network or via a computer system bus. For example, data storage managed by a DVS system can be data storage in which the DVS system has exclusive access to its data.

[0090] A "concept list" is understood to be a non-redundant, preferably sorted list of data values, where each concept list represents a specific semantic concept and contains only the data values ​​to which this semantic concept was assigned during import. The object ID assigned to a specific data value or list element in a concept list contains only the object ID of the data object to which this semantic concept was assigned during import, including that data value or list element.

[0091] A “list without a concept” is understood here to mean a non-redundant, preferably sorted list of data values ​​that do not represent any semantic concepts and include data values ​​to which no semantic concepts were assigned during import. An object ID assigned to a particular data value or list element in a list without a concept includes only the object IDs of data objects that include that data value or list element and to which no semantic concepts were assigned during import.

[0092] "Global list" is understood here as a non-redundant, preferably sorted list of data values, where each data value in the global list is assigned one or more pointers, each pointer pointing to an element of a concept list, or an element of a non-concept list containing the same data value as this data value in the global list.

[0093] Here, "semantic concept" is understood as the meaning that can be realized in various specific embodiments. For example, a semantic concept can represent a class of objects or processes that include several elements. Depending on the type of raw data and / or parser used, various different concepts may be assigned to parsed and imported data values. For example, in the case of medical data, a semantic concept could be "diagnosis," which includes various specific embodiments (data values) such as "diabetes," "Parkinson's disease," and "skin cancer." Another semantic concept could be "symptoms," which could include specific embodiments or data values ​​such as "fever," "chills," and "toothache."

[0094] In this context, "importing" parsing results by a DVS system is understood as the process by which the DVS system receives parsing results from one or more external parsers. If one or more parsers are internal components of the DVS system, importing refers to transferring these parsers or the parsing results from them to a module in the DVS system responsible for storing the results in a non-redundant list format.

[0095] Here, "computer system" is understood to mean a monolithic or distributed data processing system, particularly a digital data processing system. Therefore, a data processing system can consist of, for example, a standalone computer system or a computer network, especially a cloud system. A computer system can also be designed as, for example, a mobile data processing system, such as a notebook computer, tablet computer, or portable telecommunications device, such as a smartphone.

[0096] Here, "system" is understood to mean the totality of one or more elements capable of processing data. For this purpose, system components exchange data and / or control commands. For example, a system may include a computer having a DVS system. Optionally, a system may include further components, such as one or more client computer systems that send search queries and / or analysis queries to the DVS system, and / or source systems that each contain a portion of the raw data that is converted into a non-redundant data value list by the DVS system and parser. [Brief explanation of the drawing]

[0097] Embodiments of the present invention will be described below with reference to the drawings. [Figure 1] Figure 1 is a block diagram of a system that includes a DVS system. [Figure 2] Figure 2 is a flowchart illustrating the method for saving data. [Figure 3] Figure 3 shows multiple data structures having raw data with different structures and contents. [Figure 4] Figure 4 shows a back diagram of a portion of the list generated by the DMS system. [Figure 5] Figure 5 shows a distributed computing system with a DVS system client. [Figure 6] Figure 6 shows a screenshot of the GUI of a DVS system that displays search results using regular expressions. [Figure 7] Figure 7 shows screenshots of the GUI of a DVS system with different search options. [Figure 8] Figure 8 shows a screenshot of the DVS system's GUI, which summarizes the contents of the concept list. [Modes for carrying out the invention]

[0098] Figure 1 is a block diagram of a system 100 having a DVS system 102 according to one embodiment of the present invention.

[0099] For example, a method for storing and retrieving data as described in the flowchart of Figure 2 can be performed by the system 100 shown in Figure 1. Therefore, the system 100 and the method in Figure 2 will be described together below with reference to the two figures. For example, the system and method can be used to integrate large amounts of heterogeneously structured raw data 112 and make it searchable and analyzable in an efficient, flexible, and scalable manner.

[0100] For example, raw data may include XML files, JSON files, text files in various formats (e.g., Microsoft Word, OpenOffice, Adobe Acrobat PDF files, TXT files, etc.) of various content and structure, various tables from one or more different relational databases, media data, or hierarchically organized data, such as object trees. In some embodiments, raw data may also be input via the user interface of the DVS system. Generally, raw data originates from multiple independent data sources, such as different organizations, particularly corporations, research institutions, and public institutions. In some raw data, data objects, their data values, and optionally their semantic meanings can be more or less explicitly identified. Examples include database tables, Excel files, and other relatively highly structured data with corresponding fields. In other raw data (e.g., image data), data objects and their data values ​​may be implicit rather than explicit; that is, data objects and their data values ​​are only recognized and extracted during the parsing process.

[0101] Multiple different parsing and / or semantic parsers 118–130 are provided for parsing various raw data. For example, a parser may be part of the integration module 110 of the DVS system 102. Alternatively, a parser may be an external component whose execution is initiated and controlled by the DVS system. To parse the raw data, the parser 118–130 used requires at least read access to each piece of raw data. During parsing, for example, when dynamically creating an object ID for a data object, it is preferable for the parser to be swapped with the DVS system or other components of the DVS system so that it can determine whether this ID is truly unique or whether it is already occupied by an object ID used as a reference in one of the data value lists.

[0102] In the first step 202, the DMS system 102 and / or the parser controlled by it receive the raw data 112 or at least an access address to this raw data. The access address enables at least read access to the raw data. For example, the raw data may be generated or captured in one or more source systems connected via a network and transmitted to the computer system 100 via the network, e.g., the Internet. In some embodiments, the raw data is stored in copies by the DMS system for later processing, e.g., by additional and / or improved versions of an existing parser. However, the raw data is not processed directly to respond to a search query.

[0103] Parsing and importing different raw data can be done in parallel, but depending on the availability of the raw data and / or the parser used for parsing, it is also possible to process different raw data successively, or even at long time intervals.

[0104] In step 204, the raw data is processed by multiple parsers 118–130 to extract data objects, their object IDs, and one or more data values ​​contained within each data object. For example, different parsers may exist for Excel files, text files, image files, various XML files, and database tables. The object ID can be an identifier for a data object already provided by the raw data and adopted as the object ID by the DVS system. Alternatively, the object ID can be generated during parsing. For example, multiple data records can be stored row by row in an Excel table. The parser for this Excel table can be configured to interpret each row of the Excel file as a data object and use a combination of the file name and the row number of the row containing the dataset / data object. If a particular semantic concept has already been explicitly assigned to an extracted data value in the raw data (such as by the identifier of a corresponding field or column), the parser in charge assigns the semantic concept that the extracted data value embodies to the extracted data value. The parser uses a specific field identifier as a semantic concept, and the DVS system automatically generates a corresponding concept list. If the concept does not yet exist in data memory in the form of a concept list, the DVS system automatically learns the new semantic concept. Data values ​​extracted from the raw data during parsing are stored in the concept list according to the semantic concept assigned by the field identifier. However, in a complex parsing process, the parser may only recognize data objects and data values ​​that are not explicitly included in the raw data. Similarly, the parser may not recognize which semantic concept an extracted data value belongs to.Therefore, the result after performing the parse step will be in the form of a data value list (which may be initially redundant), where each element is linked to an object ID and optionally assigned a semantic concept.

[0105] In step 206, the DVS system imports the results of the parsing process, for example via the integration module 110, and stores the results in the form of non-redundant data value lists 114, 115, and 116 in the data storage or data memory area 104 managed by the DVS system. These lists also include one or more concept lists 116, typically a list without concepts, and optionally a global list 115. Preferably, the data values ​​in the lists are sorted according to a specific ordering relationship, for example, alphabetically. The function and structure of the lists are described in more detail with reference to Figure 4. This form of storage resolves, to some extent, the object structure, i.e., the question of which data values ​​or semantic concepts exist for a particular data object. Thus, the amount of data is very strongly compressed and can be expanded at any time without structural change by additional data values ​​and / or concepts provided in the form of additional raw data and / or additional parsers.

[0106] In step 210, the DVS system provides non-redundant lists 114, 115, and 116, responds to search queries based on these lists, and / or performs analysis on the data values ​​contained in the lists. For example, the DVS system that created the lists may include a module 106 for receiving and processing search queries, and / or a module for performing analysis. For example, module 106 may be configured to receive search queries from one or more client computers via a network interface, or directly from users working locally via a GUI. Additionally or alternatively, module 106 may also include multiple concept lists and, optionally, multiple complex analytical functions based on set operations on non-concept lists and global lists.

[0107] In other embodiments, the DMS system that created lists 114, 115, and 116 may also transfer lists 114, 115, and 116 to a search and analysis system, either as the original or as a copy. The search and analysis system may be a further instance of the DVS system, or it may be another hardware and / or software-based data processing system having an interface for receiving and processing search queries and / or analysis commands. This other search and analysis system may optionally include a parser and an interface for importing parsed results, but it is not required to include these components. Figure 3 shows a data structure with raw data, where the data structure has different structures and different contents.

[0108] For example, data structures 302, 304, and 306 are product datasheets for the engine (Motoren) in JSON format from the manufacturer (shown here as tab-delimited text files for space reasons). For example, a parsing process can be performed so that each of the three JSON files 302, 304, and 306 is interpreted as a separate data object, each having its own object ID. Each data object contains multiple key-value fields, such as specific data values ​​for the semantic concept "output" and specific data values ​​for the semantic concept "torque".

[0109] The data structure shown below is an Excel table containing specifications for various paint characteristics of a paint dealer. Each row, from 308 to 313, contains exactly one data object (data record). During the parsing process, each recognized data object can be assigned a row number in combination with an identifier in this Excel table, for example, as an object ID.

[0110] Some of the raw data may be in the form of text data, such as text files 314-318. For example, this text can be broken down into individual words (each acting as a data value) using a pure syntactic parser. For example, a syntactic parser could be a tokenisier that breaks down natural language text into words that act as data values ​​(with some stop words as needed). In this case, no semantic concepts are initially assigned to the words / data values ​​during import; instead, the words and the object IDs of their respective source objects are stored in a conceptless list, where each word appears only once.

[0111] Another portion of the raw data can be provided, for example, in the form of excerpts from commercial registers 320–324. These may contain a mixture of key-value fields and free text.

[0112] The raw data originates from different sources, some of which are redundant ("Gelb-AG"), and others whose data values ​​are ambiguous ("Silber"). Nevertheless, in this embodiment, all of this data can be efficiently integrated and processed while resolving semantic ambiguity.

[0113] Figure 4 is a block diagram showing a partial excerpt of the data structure generated by the DVS system based on the raw data shown in Figure 3. The data values ​​shown in the non-redundant lists 114, 115, and 116 are also only excerpts, and typically the lists are quite long.

[0114] All data value lists 114, 115, and 116 generated and managed by the DVS system are non-redundant, i.e., each data value is included only once. Preferably, the data values ​​are further sorted so that, based on the sorting of the search terms and the data values ​​already searched in the list, a series of searches within the list can be stopped if it is ruled that further searches within the list will not result in a hit.

[0115] For example, when the parser processed the raw data, it identified the following semantic concepts in particular: color, manufacturer, paint ID, last name, relationship type, and metal type. These semantic concepts are represented in concept lists 402, 404, 406, 408, 410, and 412, respectively.

[0116] The "No Concept" list 115 contains data values ​​to which no semantic concept was assigned during parsing. For example, if the parser of message text 314 is a pure tokenizer that converts text into words and does not recognize the meaning of words, then the words identified by this parser are "No Concept" and are stored in the "No Concept" list in a non-redundant form. Therefore, if text 314 refers to "Gelb-AG" without the parser used recognizing the semantic meaning of the word, its data value is stored in the "No Concept" list, referring to data object 314. The same data value "Gelb-AG" may also be present in other raw data, thereby making it possible to assign a semantic concept to this data value when parsing this other raw data. For example, the data value "Gelb-AG" is also present in commercial register excerpt 320 under the field identifier "Company". For example, a commercial registration parser and / or DVS system may be configured to treat the semantic concepts "company" and "manufacturer" as synonyms for the same semantic concept, and the data value "Gelb-AG" may be stored in a concept list 404 for the concept "manufacturer" associated with an object ID on data object 302. If the data value already exists in the list, only the set of object IDs assigned to this data value is appropriately supplemented.

[0117] Based not only on the concept-less list 115 but also on several other lists shown in Figure 3, it can be seen that list data values ​​are sorted by different data objects, and the sorting of data values ​​in a list is independent of the origin of the data values ​​from a particular data object. Thus, it can be seen that the original structure of the data objects contained in the raw data is completely dismantled structurally during list generation, and the assignment of data values ​​to data objects remains only in the form of object IDs, which are reconstructible.

[0118] Optionally, in some embodiments, the position data of the data values ​​within each data object can also be stored along with the object ID in Lists 115-116, and as a result, the positions of the data values ​​within the data objects can also be reconstructed using the raw data.

[0119] The non-redundant data value list preferably also includes a global list 114. This does not include object IDs and is used to assign data values ​​to one or more semantic concepts and corresponding concept lists. For example, the data value "silver" in the global list is assigned to the concept lists "color," "surname," and "metal type," while the data value "Stolze" is assigned only to the concept list "surname." For example, the global list can work to dynamically display different semantic concepts, including a given grinding term, when a search term is entered, and / or to restrict the data values ​​evaluated during database searches or data analysis to data values ​​that represent a given semantic concept.

[0120] Preferably, the list also includes one or more conceptual lists 410 representing relationships between data objects. Here, the data value consists of a konkatenation of a relationship type (e.g., "Verbaubar in (installable in ~)") and an object ID (e.g., an identifier for engine types MF-3000, ..., MF6000). This relationship specifies which motor types a given component (specified in the data object from which this relationship is extracted during parsing) can be installed on. A search query with the search value "Verbaubar_in_MF_3100 (installable in MF_3100)" quickly returns all object IDs of data objects representing engine parts installable on engines of type MF_3100. In this way, a highly efficient search function is provided for objects of as many relationship types as possible, and it can be used even by data processing devices with limited working memory and computing power.

[0121] Figure 5 shows a distributed computer system 500 having DVS system clients 506, 508 that send search queries and / or analysis queries to interface 106 of DVS system 102 via a network and receive in response a response such as a list of objects whose data values ​​satisfy the search criteria to which the queries were sent.

[0122] Raw data is provided from multiple source systems 502 and 504, for example, via a network such as the Internet.

[0123] However, the system architecture 500 shown in Figure 5 is only one of many possible architectures. Many alternative architectures are possible, for example, storing a copy of the raw data on the same computer system 100 as the DVS system, and / or using one or more local client query applications instantiated on the computer system 100 in addition to or instead of the network-connected client systems 506, 508. The parsers 118-130 may be integrated into the DVS system or may be external software programs controlled by the DVS system.

[0124] For example, parsers 118-130 can first generate a Key-Value (KV)-Listen list from the extracted information, which still includes data values ​​in a redundant format (e.g., color=green, weight=100kg, music genre=classical, book=science, etc.). Additionally or alternatively, the CT list can be received via the GUI of the DP system, in which case the parsing process only needs to accept the given key-value pairs and associate them with the object IDs of the data objects from which the key-value pairs are extracted. For example, in the GUI, a doctor can enter a diagnosis based on multiple symptoms. In this case, a "case" can be entered as a related data set (data object), and key-value pairs such as "symptom=high blood sugar," "symptom=low insulin concentration," and "symptom=dizziness," as well as "diagnosis=diabetes," are stored in the concept lists "symptoms" and "diagnosis," respectively, and the same case can be assigned in a search query via the object ID of this "case."

[0125] However, it is possible to perform comprehensive statistical analysis not only on reconstructing data values ​​for specific cases or patients, but also on lists of data values ​​including symptoms and diagnoses for hundreds of thousands of patients.

[0126] In this way, by efficiently performing correlation analysis on a list without redundancy, such as by crossing the object IDs of data values ​​from multiple concept lists, new medical insights can be obtained.

[0127] All key-value pairs 503 generated by parsers 118-130 are preferably automatically, standardized, and processed in the same way every time, and transferred to the non-redundant data-value lists 114, 115, and 116. No changes are required to the list structure or any indexes (both of which would normally have to be adapted in conventional DBMSs when integrating additional data or new datasets with previously unknown structures).

[0128] The entire list of data values ​​represents a multidimensional model in which data records do not exist as separate units structurally separated from other data records. The association between a data value and a specific data object is derived only by searching within the list without redundancy and by set operations on the object IDs of the elements in these lists.

[0129] For example, a search query could include the following search criteria for a phone book entry: "Last Name = / [CK].*ristin / ", "Street = " / .*[Aa]llee.* / ", and "Address = / 14 / ". Thus, the search query includes two regular expressions that allow "fuzzy matching" as search criteria, and the data value "14" that must be exactly met. The DVS system is configured to identify the concept list (in this case, last name, street, street, and address) to which the search query's criteria apply, and then, depending on the type of criterion, perform a fuzzy search or a search for identical values ​​in the data values ​​of each concept list. Such a search is not only very fast, but also has the advantage that phone book entries (data objects) that meet the search criteria will not be missed due to typos or spelling differences in names (provided that typos are still covered by the regular expressions).

[0130] Figure 6 shows a screenshot of the GUI600 of a DVS system having regular expression search results according to one embodiment.

[0131] In this example, an 8.9 gigabyte raw data set was parsed and transferred to a 4.42 gigabyte non-redundant list. The raw data set consisted of a collection of comma-separated files, each column containing a different semantic concept such as last name, first name, street, house number, zip code, city, area code, and telephone number. As already described in the embodiment, this data was transformed into a global list and the following concept list, each containing only non-redundant data values ​​such as first name, last name, zip code, city, house number, area code, and telephone number. Furthermore, for some data sets (data objects), relationships with other data objects (data sets) were defined, in this case, the relationship between "mother" and "father" was defined.

[0132] The GUI allows users to include search criteria related to one or more items in the list, formulate different search queries that can be used in different search modes (identity or regular expression matching), and the search mode can be selected individually for each concept index.

[0133] In the example shown in Figure 6, a regular expression is entered into search field 602 to search the non-redundant concept list "Name". In the first step, all data values ​​in the concept list "Name" that "match" the regular expression, and the object IDs assigned to these matching data values ​​in this concept list, are identified. Because this first step searches only a sorted name list with 3.2 million different data values, and not 89 gigabytes of raw data, it takes only 0.7 seconds. In the second step, data objects (data records) representing people with the "matching" surnames are output using all the object IDs identified in the first step. In the example shown here, outputting these data sets takes approximately 11 seconds. It should be noted that these execution times are based on a 2012 MacBook Air (with 8GB of RAM), where a web server and various other programs are running in addition to the DMS system for data retrieval, and the non-redundant list is stored on an external SSD storage device connected to the laptop via USB. The regular expression search was performed in less than 1 second on a laptop from several years ago, not on a dedicated database server.

[0134] Therefore, using non-redundant data value lists has the advantage of enabling highly efficient searches not only for IDs but also for regular expressions. Furthermore, this specific data organization using non-redundant data value lists allows for handling a variety of search queries, as shown in Figure 7.

[0135] Figure 7 shows a screenshot of the GUI700 of the DVS system, which has various search options. On the one hand, the user has the option to perform a "global search," which is called a "universal search." In a universal search, a search term for ID search or a search term in the form of a regular expression is entered. The system then searches a global list that contains the entirety (Gesamtheit) of all imported data values. For example, the global list contains the search term "Berlin" once, and this data value in the global list refers to multiple concept lists "First Name," "City," "Street," and "First Name," where this data value appears 1327 times, 1112234 times, 89412 times, and 81 times, respectively. On the other hand, the data value "Berlin" does not appear in the concept lists "Street Address" and "Zip Code." The fact that the last name "Berlin" is hit 81 times may indicate an error in the raw data. For example, the field assignment may not be recognized during parsing due to a missing or incorrect separator.

[0136] In addition to or instead of global searches, users can also define and perform any combination search. In a combination search, the user can enter search criteria for each semantic concept to be considered in the search (here, for example, name, personal abbreviation, and phone number or street), and preferably, each time additional characters are entered for any of the search criteria, a search is triggered against all already entered search criteria (whole or partially).

[0137] Even with several regular expressions for several concept lists, execution times range from a few seconds to tens of seconds, depending on the type of expression. It's also noteworthy that there's no need to create indexes or adjust the list structure (index) depending on the number and type of semantic concepts used for searching. Many conventional DBMSs require the creation of large search indexes in addition to tabular relational data, their size roughly equivalent to 8.9GB of raw data. Therefore, the effective memory consumption of a conventional relational DBMS supporting the same number of search queries is many times higher than the 4.42GB consumed by the DVS system in the embodiment described here, because it enables any combination of search queries possible on global lists and / or concept lists and / or lists without concepts.

[0138] In another example, a 550GB raw dataset containing 1.4 billion taxi travel records (including time and location information for origin and destination, number of passengers, and payment amount) was transformed into a non-redundant list set with a total size of 396GB. Each data record (data object) in the raw data referred to the origin and destination taxi zones. Despite the enormous size of the dataset, the execution time to determine the total number of taxi travels originating from or destined for a given zone was only 0.01 seconds, thanks to the list transformation described.

[0139] Figure 8 shows the GUI800 screen of the DVS system, where the contents of the concept lists are summarized. The column "Field name (Feldname)" contains the name of the concept list managed and used for searching by the DVS system. This name is identical to the semantic concept represented by this concept list. The column "Field index entries" shows the number of different data values ​​in each concept list. The concept list "Telephone" has 6,941,134 different telephone numbers (without area codes). The concept list "Name" has 3,210,129 different surnames. The concept list "First name" has 673,264 possible first names. The total number of (different, i.e., non-redundant) object IDs assigned to all data values ​​in each list is shown in the "Occurrences (Vorkommen)" column. Therefore, the raw data contains 34,795,673 data objects (data records) that have data values ​​for the field "Phone," and across all of these "Phone field" entries, a total of 6,941,134 different phone numbers are included and extracted.

Claims

1. A method for storing data in data storage, wherein the method is - A step of receiving raw data or an access address of the raw data by a data processing and retrieval system (DVS system), wherein the raw data has a different structure; - A step of parsing raw data using multiple different parsers, each of which identifies a data object having one or more data values ​​and respective object IDs for the data objects, wherein at least some of the data values ​​are each assigned a semantic concept; - The step of automatically importing the parsing results from the DVS system; - A step of automatically saving all parsing results in the form of a non-redundant data value list within the data storage by the DVS system, The aforementioned list without redundancy is: - A concept list comprising one or more concept lists, each of which represents one semantic concept, and a non-redundant list selectively containing imported data values ​​to which the semantic concepts of the concept list were assigned during parsing, each data value in the concept list being assigned all object IDs of the data object containing that data value, and the included data values ​​are representations of the semantic concepts of this concept list, and A step including: a conceptless list, the conceptless list selectively includes imported data values ​​to which no semantic concepts were assigned during the parsing, each data value in the conceptless list is assigned all object IDs of the data object containing that data value, and the conceptless list was unable to assign semantic concepts to the data values ​​contained in this data object during the parsing; - A step of providing the non-redundant list by the DVS system in order to respond to a search query and / or to perform data analysis, wherein the search query and / or data analysis is performed without accessing the raw data; Methods that include...

2. At least a portion of the aforementioned raw data exists or is received in the form of multiple data structures, The aforementioned multiple data structures are, - XML ​​file, - JSON file, - Text file, - CSV file, - Data bank table, - Object tree, - Media files, video, audio, image files, - Data received via GUI, - Streaming data, A mixture of two or more of the following data structures: The method according to claim 1.

3. The aforementioned list without redundancy is: - A global list, The aforementioned global list is a non-redundant list of all imported data values, Each data value in the global list is assigned one or more pointers. Each pointer includes a global list that points to one of the elements of the concept list or an element of the no-concept list that contains the same data value as that element, - The DVS system is configured to perform analysis or query searches on at least the global list, The aforementioned global list is used to recognize and / or process concept-specific data values ​​that represent different semantic concepts within different data objects. The method according to claim 1 or 2.

4. - At least a portion of the aforementioned data values ​​are extracted from the fields of the data structure that constitute the raw data, The aforementioned fields are defined by the aforementioned data structure, The field includes one or more concept-related fields and / or one or more concept-unknown fields. Concept-related fields are fields to which a field identifier has been assigned. The aforementioned field identifier represents a semantic concept; and / or At least some of the data values ​​are imported by a semantic parser that recognizes the imported data values ​​and the semantic concepts assigned to them based on the data analysis. - Based on data analysis, at least a portion of the imported data values ​​are imported by a semantic parser that recognizes the imported data values ​​and the semantic concepts assigned to them. The aforementioned data analysis includes image analysis, audio signal analysis, statistical analysis, classification methods, machine learning methods, and / or pattern recognition methods. - The DVS system stores the data values ​​extracted from concept-related fields in one of the concept lists representing the semantic concepts of the field identifiers of those concept-related fields; and / or - The DVS system, when the parser used does not recognize the semantic concepts of the data values, stores the data values ​​extracted from concept-unknown fields in a dedicated concept-unknown list. The method according to any one of claims 1 to 3.

5. - A further step of providing a parser designed to recognize and import data values ​​that have been imported and stored, and which have been assigned to at least one new semantic concept, The aforementioned new semantic concept is a concept that is not represented by any of the concepts currently included in the data storage, namely, Step, - A step of processing the raw data with a further parser, Steps include: extracting one or more new data values ​​from raw data assigned to at least one new concept; - Compare at least one new semantic concept recognized by the aforementioned further parser with the list of concepts in the data memory of the DVS system, The steps include: automatically generating and storing a new concept list for each of the at least one new semantic concepts from which the further parser has extracted at least one data value; - A step of automatically saving the data values ​​extracted from the raw data by the further parser to at least one new concept list representing a new semantic concept assigned to the data values ​​by the further parser, The method according to any one of claims 1 to 4, further comprising:

6. - A step of receiving a search query to identify a data object that satisfies one or more concept-related search criteria and / or one or more concept-unknown search criteria, wherein the concept-related search criteria are search criteria to which a semantic concept identifier is assigned, and the concept-unknown search criteria are search criteria to which the semantic concept is not assigned, - A step of searching the no-concept list for one or more data values ​​that satisfy each of the received concept-unknown search criteria, and / or, - For each of the received concept-related search criteria, the step of selectively searching for one or more data values ​​that satisfy the concept-related search criteria from the concept list representing the semantic concept of the search criterion, - In response to a search query by the DVS system, the steps include returning an object ID assigned to a data value identified during a search of the global list and / or at least one of the concept lists, or a subset of such object IDs; The method according to any one of claims 1 to 5, further comprising:

7. Using the non-redundant list to perform query searches and / or data analysis is, - A step of performing a set operation on a set of object IDs assigned to two or more data values ​​in the non-redundant list, The set operation includes operations on intersection sets, join sets, difference sets, or symmetric difference sets. The method according to any one of claims 1 to 6, further comprising:

8. - At least some of the data objects are related to other data objects; - The DVS system, in the process of analyzing raw data, and / or in the process of subsequent data processing operations performed on raw data and / or data values ​​in already stored non-redundant lists, Extract object relationships between these data objects and other data objects. - Each extracted relationship between one of the data objects (referred to as the first data object) and another of the data objects (referred to as the second data object) is extracted in the form of a combination of relationship type and the object ID of the second data object. - Each of the extracted combinations is stored as one of the data values ​​in the no-concept list and / or at least one of the concept lists. In the aforementioned list of no concepts and / or at least one list of concepts, Each of the extracted combinations is assigned the object ID of the first data object in which the relationship specified by the combination exists. The method according to any one of claims 1 to 7.

9. - A step of receiving a search query or analysis command to identify all data objects that have a relationship with a search data object, wherein the query or command includes a search object ID, - A step of searching a global list to identify all data values ​​that consist of a combination of a relationship type and an object ID identical to the search object ID, - The steps of identifying one or more concept lists that refer to the data value determined in the global list; - A step of searching the one or more identified concept lists to identify a data value consisting of a combination of the relation type and an object ID identical to the search object ID, - A step of returning the object ID assigned to the identified data value in one or more concept lists, i.e., the ID of the data object that has a relationship with the search object, The method according to claim 8, further comprising:

10. - A step of providing a mapping table by the DVS system, wherein the mapping table assigns one or more identifiers to each data value in the no-concept list and each data value in the concept list, and the identifiers are specific values ​​whose length and / or type depends on the processor architecture of the computer system used for retrieval and / or analysis. - A step of generating obfuscated copies of the concept-less list and each concept list using a DVS system, wherein in the obfuscated copies, each data value is replaced with an identifier assigned to the data value in the mapping table, - A step of using the obfuscated list to perform a search query and / or analysis using the DVS system, The method according to any one of claims 1 to 9, further comprising:

11. - The step of encrypting the concept list and the no-concept list using the DVS system, or the step of encrypting the obfuscated concept list and the obfuscated no-concept list using the DVS system; - In response to receiving the search query or data analysis command, the step of identifying the list from the encrypted list that needs to be processed in order to process the search query or data analysis command; - A step of authenticating the sender of a search query to a DVS system, wherein the DVS system individually checks, for each identified list, whether the sender has permission to access the contents of the list; - The DVS system generates selectively decrypted copies from a list that the sender is authorized to access; - Perform the search query or data analysis on the decrypted list copy. - After executing the search query or the data analysis, the DVS system deletes the decrypted list copy, and the DVS system uses a different encryption key for encrypting each list, step, The method according to any one of claims 1 to 10, further comprising:

12. - At least one of the above concept lists represents a genome marker for multiple individuals of a species or subspecies, At least one of the aforementioned concept lists represents the metabolic or phenotypic characteristics of these individuals, The DVS system performs data analysis using set operations on lists, and The aforementioned data analysis may include correlation analysis of genomic markers on the one hand, and metabolic or phenotypic traits on the other hand, or - At least a portion of the data values ​​in the global list and / or at least one of the concept lists specify an "ist-Bestandteil-von" relationship with other objects, including the object ID of the other object. At least one of the aforementioned concept lists represents the metabolic or phenotypic characteristics of these individuals, The DVS system performs a query search, The search criteria include a combination of the relationship type "ist-Bestandteil-von" and the object ID. The object ID of the search criterion represents a machine or vehicle or one of the multiple components of that machine or vehicle, or - At least some of the data values ​​in the global list and / or at least one of the concept lists are natural language words, The DVS system performs data analysis using set operations on lists, and The aforementioned data analysis involves word correlation analysis to create this natural language prediction model. The method according to any one of claims 1 to 11, further comprising:

13. The aforementioned data values ​​are sorted in the aforementioned list without redundancy. The method according to any one of claims 1 to 12, further comprising:

14. A volatile or non-volatile storage medium in which computer-readable instructions are stored, wherein the instructions are designed to cause a processor to execute the method according to any one of claims 1 to 13, which involves storing data in a data memory. storage medium.

15. A computer system for storing data, - At least one processor, - Data storage and, - A data processing and retrieval system (DVS system) is provided, The DVS system is designed to manage and retrieve data stored in the data storage. The aforementioned at least one processor, - Data processing and retrieval system - A step of receiving raw data or an access address of said raw data by a DVS system, wherein the said raw data has a different structure; - A step of parsing raw data using multiple different parsers, each of which identifies a data object having one or more data values ​​and respective object IDs for the data objects, wherein at least some of the data values ​​are each assigned a semantic concept; - The step of automatically importing the parsing results from the aforementioned DVS system; - The DVS system automatically saves all parsing results in the form of a non-redundant data value list within the data storage; The aforementioned list without redundancy is: ...One or more concept lists, each of which represents one semantic concept, and a non-redundant list selectively includes imported data values ​​to which the semantic concepts of the concept lists were assigned during the parsing process, and each data value in the concept list is assigned all object IDs of the data object containing that data value, and the included data values ​​are representations of the semantic concepts of this concept list, and a concept list, ...a conceptless list, which selectively includes imported data values ​​to which no semantic concepts were assigned during the parsing, and each data value in the conceptless list is assigned all object IDs of the data object containing that data value, and to which no semantic concepts could not be assigned during the parsing; - The DVS system provides the non-redundant list in order to respond to a search query and / or to perform data analysis; It is designed to perform a method that includes, Computer system.

16. The at least one processor includes an arithmetic logic unit (ALU), the ALU is It is designed to perform set operations on a set of object IDs assigned to two or more data values ​​in a list without redundancy. Set operations include operations on intersection sets, join sets, difference sets, or symmetric difference sets. The computer system according to claim 15.

Images