Unstructured data analysis in legacy data warehouses

Abstract

A method (500) for unstructured data analysis includes receiving an unstructured data query (20A) from a user (12) that requests data processing hardware (142) to determine one or more unstructured data files (172) stored at a data repository (170) that match query parameters (21). The method includes determining, using an object table (352), a set of unstructured data files (172) stored at the data repository that match the query parameter. The object table includes: a plurality of rows, each row of the plurality of rows being associated with a respective unstructured data file stored at the data repository; and a plurality of columns, each column of the plurality of columns including metadata (305) associated with the respective unstructured data file of each row of the plurality of rows. The method includes returning a structured data table including the determined set of unstructured data files to the user (252).

Description

Technical Field

[0001] The present disclosure relates to unstructured data analysis in traditional data warehouses. Background Art

[0002] Analyzers generally refer to tools that can be used to process large collections of data, such as data stored in traditional data warehouses. While these analyzers are useful for managing, governing, securely sharing / storing, and analyzing big data, they tend to be strict about the types of data that can be used. For example, some analyzers implement structured query language (SQL) to process data and are therefore only configured to process structured data. These example analyzers cannot be used to process unstructured data without additional manual conversion of the unstructured data into structured data. Summary of the invention

[0003] One aspect of the present disclosure provides a method for unstructured data analysis in a traditional data warehouse. The method, when executed by data processing hardware, causes the data processing hardware to perform an operation, the operation comprising receiving an unstructured data query from a user, the unstructured data query requesting the data processing hardware to determine one or more unstructured data files stored at a data repository that match the query parameters. The operation also includes using an object table to determine a set of unstructured data files stored at the data repository that match the query parameters. The object table includes: a plurality of rows, each of the plurality of rows being associated with a corresponding unstructured data file stored at the data repository; and a plurality of columns, each of the plurality of columns including metadata associated with a corresponding unstructured data file of each of the plurality of rows. The operation further includes returning a structured data table including a set of determined unstructured data files to the user.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the unstructured data includes image or audio data. Determining the set of unstructured data files may include, for each unstructured data file of the object table, determining that the corresponding unstructured data file matches the query parameter based on metadata of the corresponding unstructured data file, and adding the corresponding unstructured data file to the set of unstructured data files based on determining that the corresponding unstructured data file matches the query parameter.

[0005] In some implementations, the operation further includes generating the object table using the machine learning model prior to receiving the unstructured data query from the user. In these implementations, generating the object table may include selecting one or more unstructured data files for the object table based on the object table query. Generating the object table also includes generating a plurality of rows based on a number of the one or more unstructured data files. Further, generating the object table includes determining metadata included in the object table using the machine learning model, and generating a plurality of columns based on the metadata. Generating the object table also includes populating the object table with references to the one or more unstructured data files and metadata associated with each corresponding unstructured data file.

[0006] In some implementations, the operation further includes generating the object table using a data scraper prior to receiving the unstructured data query from the user. In other implementations, the metadata includes at least one of a number of bytes, a file type, a creation time, a location, or business metadata. At least one row of the object table may be associated with a row access policy that restricts access to the corresponding unstructured data file. In some implementations, the operation further includes periodically updating the object table based on changes at the data repository. In these implementations, the object table may be updated based on a refresh rate.

[0007] Another aspect of the present disclosure provides a system for unstructured data analysis in a traditional data warehouse. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. The operation includes receiving an unstructured data query from a user, the unstructured data query requesting the data processing hardware to determine one or more unstructured data files stored at a data repository that match the query parameters. The operation also includes using an object table to determine a collection of unstructured data files that match the query parameters stored at the data repository. The object table includes: a plurality of rows, each of which is associated with a corresponding unstructured data file stored at the data repository; and a plurality of columns, each of which includes metadata associated with a corresponding unstructured data file of each of the plurality of rows. The operation further includes returning a structured data table including a collection of determined unstructured data files to the user.

[0008] This aspect may include one or more of the following optional features. In some implementations, the unstructured data includes image or audio data. Determining the set of unstructured data files may include, for each unstructured data file of the object table, determining that the corresponding unstructured data file matches the query parameter based on metadata of the corresponding unstructured data file, and adding the corresponding unstructured data file to the set of unstructured data files based on determining that the corresponding unstructured data file matches the query parameter.

[0009] In some implementations, the operation further includes generating the object table using the machine learning model prior to receiving the unstructured data query from the user. In these implementations, generating the object table may include selecting one or more unstructured data files for the object table based on the object table query. Generating the object table also includes generating a plurality of rows based on a number of the one or more unstructured data files. Further, generating the object table includes determining metadata included in the object table using the machine learning model, and generating a plurality of columns based on the metadata. Generating the object table also includes populating the object table with references to the one or more unstructured data files and metadata associated with each corresponding unstructured data file.

[0010] In some implementations, the operation further includes generating the object table using the data crawler prior to receiving the unstructured data query from the user. In other implementations, the metadata includes at least one of a number of bytes, a file type, a creation time, a location, or business metadata. At least one row of the object table may be associated with a row access policy that restricts access to the corresponding unstructured data file. In some implementations, the operation further includes periodically updating the object table based on changes at the data repository. In these implementations, the object table may be updated based on a refresh rate.

[0011] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram of an example object table system for unstructured data analysis in a traditional data warehouse.

[0013] Figure 2 is a schematic diagram of an example organizer module that generates one or more structured tables.

[0014] Figure 3 is a diagram of an example object table.

[0015] Figure 4Ais a schematic diagram of an example structured table viewed by a first user.

[0016] Figure 4B is a schematic diagram of an example structured table viewed by a second user.

[0017] Figure 5 is a flow diagram of an exemplary operational arrangement of a method for unstructured data analysis in a traditional data warehouse.

[0018] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0019] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0020] Analytical programs, such as those that implement structured language queries (SQL), can be used to handle structured data, such as numbers and strings, or large collections of data that can be represented in rows and columns in a relational database. Databases are typically not configured to handle unstructured data (e.g., images, audio, video, word processing files, emails, spreadsheets), which often cover the majority of data available to users. Integrating unstructured data for analytical programs using conventional methods involves manual entry and other inflexible solutions. For example, preparing unstructured data for analytical programs includes extracting structured entities from the unstructured data using machine learning models. However, this extraction process comes with high implementation costs and time-to-value costs. Further, lineage information between unstructured data and the extracted structured data is difficult to maintain, creating additional obstacles.

[0021] The implementation of this article provides for unstructured data in an analysis program, which is configured to process structured data by using an object table. As used herein, an object table is an index of unstructured data that can be used by a downstream analysis or processing program. In some implementations, an object table is a collection of files (i.e., unstructured data) stored or referenced in a table database. For example, an object table includes multiple rows and columns, each of which corresponds to a single unstructured data file, and each column corresponds to metadata related to the file (such as metadata extracted from the file header of the file). In some implementations, the analysis program can then ingest unstructured data via an object table as structured data. In other implementations, the analysis program can infer one or more structured tables from the object table, and then use one or more structured tables as structured data. Therefore, the analysis program can process unstructured data as if the unstructured data is structured data (i.e., all analysis program processing tools and security features can be applied to unstructured data).

[0022] refer to Figure 1 In some implementations, the example object table system 100 includes a remote system 140 that communicates with one or more user devices 10 via a network 112. The remote system 140 (also referred to herein as a cloud computing environment 140) can be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic resources 142, which include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). A data repository 150 (i.e., a remote storage device) can be overlaid on the storage resources 146 to allow one or more of the clients (e.g., user devices 10) or computing resources 144 to scalably use the storage resources 146. The data repository 150 is configured to store structured data 152, structured data tables 252, and one or more object tables 352. The data repository 150 can store any number of structured data 152, structured data tables 252, and object tables 352 at any point in time. The data lake 170 may be part of the remote system 140 or otherwise communicatively coupled to the remote system 140. The data lake 170 may store unstructured data files 172, such as audio, video, images, text documents, spreadsheets, social media content, emails, and the like.

[0023] The remote system 140 is configured to receive unstructured data queries 20, 20A and / or object table queries 20, 20B from a user device 10 associated with a corresponding user 12 via, for example, a network 112. The user device 10 may correspond to any computing device, such as a desktop workstation, a laptop workstation, or a mobile device (i.e., a smart phone). The user device 10 includes computing resources 18 (e.g., data processing hardware) and / or storage resources 16 (e.g., memory hardware). The user 12 may use an analysis program 14 (e.g., via an analysis program executed on the user device 10 and / or the remote system 140) to construct an unstructured data query 20A and / or an object table query 20B. Each query 20 requests that one or more unstructured data files 172 (or a reference to one or more unstructured data files 172) be returned to the user device 10. In some examples, the unstructured data query 20A requests the remote system 140 to return a structured data table 252 based on the object table 352. The structured data table 252 is a table that includes unstructured data files 172 (and / or references to unstructured data files 172, such as links to locations where unstructured data files 172 are stored) in a form that can be ingested by the analysis program 14. The structured data table 252 can be generated based on the unstructured data query 20A with data extracted from the object table 352 (i.e., unstructured data files 172). The object table 352 is a larger data table that includes unstructured data files 172 (or references to unstructured data files 172) from the data lake 170. In some implementations, the object table 352 is generated in response to the object table query 20B. Alternatively, the object table 352 can be pre-generated. The object table 352 can include each unstructured data file 172 in the data lake 170 (or a reference to each unstructured data file 172). In other implementations, the object table 352 includes a subset of the unstructured data files 172 in the data lake 170. Both the structured data table 252 and the object table 352 are tables that include unstructured data files 172 that are organized so that they can be ingested by the analysis program 14 (e.g., by being in table form). The tables 252, 352 may include actual data from the unstructured data files 172 in the data lake 170 and / or references to the unstructured data files 172 (so that the analysis program 14 can retrieve the unstructured data files 172 from the data lake 170). The unstructured data files 172 may refer to any unstructured data, such as audio, video, images, emails, spreadsheets, text documents, etc.

[0024] The remote system 140 executes an organizer module 260, which includes a table module 262 and an inference module 264. The organizer module 260 generates one or more structured data tables 252 in response to the unstructured data query 20A and provides the one or more structured data tables to the user device 10. The table module 262 can generate an object table 352 (e.g., in response to the object table query 20B), which stores or references the unstructured data files 172 and corresponding information (e.g., metadata 305 ( Figure 3 )). The organizer module 260 accesses the object table 352 based on the received query parameters (i.e., from the unstructured data query 20A) to return the desired results. In some implementations, the table module 262 periodically updates the object table 352 and / or the structured data table 252 based on the data that has changed, added, or deleted in the data lake 170. The table module 262 can be a machine learning model trained to select unstructured data files 172 and corresponding metadata 305 for the object table 352 and / or the structured data table 252. In some implementations, the table module 262 is a data crawler that selects some or all of the unstructured data files 172 and crawls the metadata 305 from the headers of the corresponding unstructured data files 172 to populate the object table 352 and / or the structured data table 252.

[0025] In some implementations, the object table 352 includes or references at least a portion of the unstructured data files 172 stored in the data lake 170. After receiving the unstructured data query 20A, the reasoning module 264 can use the object table 352 and the query 20A (e.g., the query parameters of the query 20A) to generate one or more structured data tables 252 for use by the analysis program 14. For example, the unstructured data query 20A includes query parameters in the form of one or more SQL statements (e.g., select all .jpg files) that indicate a specific file type or other criteria based on the metadata 305 included in the unstructured data files 172. In other implementations, the analysis program 14 receives and processes the entire object table 352.

[0026] The structured data 152, the structured data tables 252, and the object data tables 352 can all be stored in the data repository 150 of the cloud computing environment 140. Here, the structured data 152 is any conventional structured data that can be ingested by the analysis program 14 without further modification. In some examples, the structured data tables 252 and the object tables 352 include attributes that are not conventionally applicable to the unstructured data files 172. For example, the structured data tables 252 can be transmitted securely and can also include customized access settings for each unstructured data file 172 in the object tables 352. For example, as discussed in more detail below, the remote system 140 implements one or more row access control policies for the object tables 352 that restrict access to the unstructured data files 172 in a manner that is not feasible when directly restricting access to the unstructured data files 172.

[0027] Figure 2 200 is a schematic diagram of an example of an organizer module 260 that generates one or more structured tables 252. As described above, the organizer module 260 can receive an unstructured data query 20A or an object table query 20B. The unstructured data query 20A can include query parameters 21, 21A-21B, such as data types 21, 21A and / or criteria 21, 21B. The data type 21A can indicate which unstructured data files 172 to select from the object table 352 for the structured data table 252. For example, the data type 21A simply indicates to select all JPEG files in the object table 352. In another example, the data type 21A can indicate a specific customer. Here, the table module 262 is a machine learning model trained in selecting data for a specific customer and thus generating a structured table 252. The table module 262 may be a machine learning model, a data crawler, or any other suitable module that may generate an object table 352 and / or a structured table 252 including an unstructured data file 172 and related metadata 305 .

[0028] The criterion 21B may be an indication that the unstructured data file 172 is to be placed in the structured data table 252. Therefore, the reasoning module 264 may generate one or more structured data tables 252 from the object table 352 based on the criterion 21B. For example, the reasoning module 264 checks each unstructured data file 172 of the object table 352 to determine whether the unstructured data file 172 matches the criterion 21B. When the unstructured data file 172 matches the criterion 21B, the reasoning module 264 places the unstructured data file 172 in the structured data table 252. The reasoning module 264 traverses the entire object table 352 to select a set of unstructured data files 172 that satisfy the query parameter 21 for the structured data table 252.

[0029] The organizer module 260 may receive a refresh rate 22. The refresh rate 22 may indicate a frequency of updating the object table 352 or the structured data table 252. For example, based on the refresh rate 22, when it is determined that one or more unstructured data files 172 have been changed, added, or deleted in the data lake 170, the organizer module 260 periodically updates the object table 352 or the structured data table 252. The organizer module 260 may implement a table module 262 and / or an inference module 264 to add or remove corresponding rows in the corresponding tables 352, 252. In some implementations, the refresh rate 22 is a preset value. Here, the refresh rate 22 may be modified by user input after the structured data table 252 is created or at a later time.

[0030] In some implementations, the object table 352 is pre-generated based on the data lake 170. For example, the object table 352 includes each unstructured data file 172 in the data lake 170. Alternatively, the object table 352 includes each unstructured data file 172 of a certain type based on the metadata 305 (e.g., all files of a certain type based on creation time, based on size). In some implementations, the object table 352 is generated based on the object table query 20B. For example, the object table query 20B may include an instruction to create the object table 352, wherein the query parameter 21 indicates the unstructured data files 172 from the data lake 170 to be included in the object table 352. The object table query 20B may include the data lake specifications 21, 21C as the query parameters 21, and any query parameters in the query parameters 21 mentioned above. In some implementations, the query 20 and the corresponding query parameters 21 and / or the refresh rate 22 are received as a script implemented in a programming language (e.g., SQL). Using the table module 262 and the reasoning module 264 , the organizer module 260 can generate an object table 352 and / or one or more structured data tables 252 based on the script in the unstructured data query 20 .

[0031] The above example query 20 is not intended to be limiting, and the query 20 may include any relevant instructions for generating the structured data table 252 and / or the object table 352 .

[0032] Reference now Figure 3, schematic diagram 300 includes an example object table 352, 352A. The example object table 352A includes four unstructured data files 172, 172A-172D, each of which corresponds to a respective row of the object table 352A. Each column of the example object table 352A corresponds to metadata 305 or row access policy 310 (included as a column of the object table 352A for illustrative purposes). Thus, each cell of the example object table 352A includes metadata 305 corresponding to a respective unstructured data file 172. The example object table 352A is not intended to be limiting, and may include any suitable number of rows and columns for generating the object table 352 based on the number of unstructured data files 172 and the different segments of metadata 305, respectively. The metadata 305 may include any information that can be retrieved, exported, or determined for the corresponding unstructured data file 172, such as the file name, byte count, file type, creation time, location, business metadata (e.g., specific metadata added to the file by the enterprise), classification data (e.g., generated by a machine learning model), user-provided data, tags, etc.

[0033] By organizing the unstructured data files 172 into a table (e.g., an object table 352 or a structured data table 252) and storing the table in the cloud computing environment 140, the system can apply features that are typically reserved for structured data in an analytics environment to the unstructured data files 172. For example, when the unstructured data files 172 are retrieved from the data lake 170, access to each unstructured data file 172 is binary (i.e., yes, access to the file is possible, or no, access to the file is not possible). In addition, by organizing the unstructured data files 172 into a structured form (e.g., an object table 352 or a structured data table 252), each unstructured data file 172 can have a specific row access policy 310 that controls access to the unstructured data file 172 at the row or even column level. That is, when the user 12 does not have access to at least a portion of the rows of the object table 352, the remote system 140 may prohibit access to the protected data or even to the unstructured data file 172 itself.

[0034] Reference now Figure 4A , schematic 400A includes an example structured table 252, 252A. In this example, structured table 252A includes columns indicating the file name of each unstructured data file 172, the creation time of each unstructured data file 172, the file type of each unstructured data file 172, and the row access strategy 310 for each row of table 252A. Here, reasoning module 264 obtains the row access strategy 310 from Figure 3An example structured table 252A is generated from an example object table 352A of . For example, the structured data table 252A may include any unstructured data files 172 extracted from the object table 352A, thereby including corresponding row access policies 310 and metadata 305 for each unstructured data file 172. In this example, a first user 12, 12A viewing the structured data table 252A has a first credential 410, 410A that is equivalent to "User A". Therefore, the first user 12A lacks access to specific cells of the structured data table 252A (as indicated by the blacked-out cells) because the first user 12 does not have credentials 410A sufficient to access those cells. For example, because the credentials 410A of the first user 12A are not equivalent to "Administrator", the first user 12A cannot access the second row of the table 252A at all. Similarly, because the credentials 410A of the first user 12A are not equivalent to "User C," the first user 12A lacks access to a portion of the third row of the table 252A.

[0035] Reference now Figure 4B , schematic 400B includes the same example structured table 252A. Here, the second user 12, 12B has different access credentials than the first user 12A, and therefore has different access rights to the unstructured data files 172 of the example structured data table 252A. Specifically, the second user 12B has credentials 410, 410B equivalent to "Administrator B", and therefore has access rights to the second row of the table 252A (i.e., because the second user 12B has administrator-level access rights). However, because the second user 12B does not have credentials 410B equivalent to "User C", the second user 12B also lacks access rights to the portion of the third row of the table 252A.

[0036] The example structured data table 252A is not intended to be limiting and may include any suitable number of rows and columns for generating the object table 352 based on the number of unstructured data files 172 and the different pieces of metadata 305, respectively. For example, the structured data table 252 may include any data extracted from the object table 352. The metadata 305 may include any information that can be retrieved, derived, or determined for the corresponding unstructured data file 172, such as the file name, number of bytes, file type, creation time, location, business metadata (e.g., specific metadata added to the file by the enterprise), classification data (e.g., generated by a machine learning model), user-provided data, tags, etc. Similar row access controls may alternatively or additionally be applied to the object table 352.

[0037] Figure 51 is a flow chart of an exemplary operational arrangement of a method 500 for unstructured data analysis in a traditional data warehouse. The data processing hardware 142 of the remote system 140 may execute instructions stored on the memory hardware 144, which cause the data processing hardware 142 to perform the operations of the method 500. The method 500 may be performed by Figure 1 The object table system 100 and / or Figure 6 Executed by various elements of the computing device 600 of the embodiment of the present invention. At operation 502, the method 500 includes receiving an unstructured data query 20A from a user 12, the unstructured data query 20A requesting the data processing hardware 144 to determine one or more unstructured data files 172 stored at a data repository 170 (e.g., a data lake 170) that match the query parameters 21 of the query 20A. At operation 504, the method 500 includes using the object table 352 to determine a set of unstructured data files 172 stored at the data repository 170 that match the query parameters 21. The object table 352 includes: a plurality of rows, each of the plurality of rows being associated with a corresponding unstructured data file 172 stored at the data repository 170; and a plurality of columns, each of the plurality of columns including metadata 305 associated with the corresponding unstructured data file 172 of each of the plurality of rows. At operation 506 , the method 500 further includes returning the structured data table 252 including the determined set of unstructured data files 172 to the user 12 .

[0038] Figure 6 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0039] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to the high-speed interface 640. In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 600 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0040] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0041] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0042] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such a distribution of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0043] The computing device 600 can be implemented in many different forms, as shown. For example, it can be implemented as a standard server 600a or multiple times as a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0044] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpreted on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0045] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0046] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0047] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The process and logic flow can also be performed by special-purpose logic circuit systems, such as FPGA (field programmable gate array) or ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. In general, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk or an optical disk, or be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer does not have to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0048] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on a client device of the user in response to a request received from the web browser.

[0049] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (500) which, when executed by data processing hardware (142), causes the data processing hardware (142) to perform operations comprising: receiving an unstructured data query (20A) from a user (12), the unstructured data query (20A) requesting the data processing hardware (142) to determine one or more unstructured data files (172) stored at a data repository (170) that match a query parameter (21); A set of unstructured data files (172) stored at the data repository (170) that match the query parameters (21) is determined using an object table (352), the object table (352) comprising: a plurality of rows, each row of the plurality of rows being associated with a corresponding unstructured data file (172) stored at the data repository (170); and a plurality of columns, each column of the plurality of columns comprising metadata (305) associated with the corresponding unstructured data file (172) for each row of the plurality of rows; and A structured data table (252) including the determined set of unstructured data files (172) is returned to the user (12).

2. The method (500) of claim 1, wherein the one or more unstructured data files (172) include image or audio data.

3. The method (500) of claim 1 or claim 2, wherein determining the set of unstructured data files (172) comprises, for each unstructured data file (172) of the object table (352): determining, based on the metadata (305) of the corresponding unstructured data file (172), that the corresponding unstructured data file (172) matches the query parameter (21); and Based on determining that the corresponding unstructured data file (172) matches the query parameter (21), the corresponding unstructured data file (172) is added to the collection of unstructured data files (172).

4. The method (500) of any one of claims 1 to 3, wherein the operation further comprises generating the object table (352) using a machine learning model before receiving the unstructured data query (20A) from the user (12).

5. The method (500) of claim 4, wherein generating the object table (352) comprises: selecting the one or more unstructured data files (172) for the object table (352) based on an object table query (20B); generating the plurality of rows based on a quantity of the one or more unstructured data files (172); determining the metadata (305) included in the object table (352) using the machine learning model; generating the plurality of columns based on the metadata (305); and The object table (352) is populated with references to the one or more unstructured data files (172) and the metadata (305) associated with each corresponding unstructured data file (172).

6. The method (500) of any one of claims 1 to 5, wherein the operation further comprises generating the object table (352) using a data crawler before receiving the unstructured data query (20A) from the user (12).

7. The method (500) of any one of claims 1 to 6, wherein the metadata (305) comprises at least one of a number of bytes, a file type, a creation time, a location, or business metadata (305).

8. The method (500) of any one of claims 1 to 7, wherein at least one row of the object table (352) is associated with a row access policy (310) that restricts access to the corresponding unstructured data file (172).

9. The method (500) of any one of claims 1 to 8, wherein the operations further comprise periodically updating the object table (352) based on changes at the data repository (170).

10. The method (500) of claim 9, wherein the object table (352) is updated based on a refresh rate (22).

11. A system (100), comprising: Data processing hardware (142); as well as Memory hardware (144) in communication with the data processing hardware (142), the memory hardware (144) storing instructions that, when executed on the data processing hardware (142), cause the data processing hardware (142) to perform operations, the operations comprising: receiving an unstructured data query (20A) from a user (12), the unstructured data query (20A) requesting the data processing hardware (142) to determine one or more unstructured data files (172) stored at a data repository (170) that match a query parameter (21); A set of unstructured data files (172) stored at the data repository (170) that match the query parameters (21) is determined using an object table (352), the object table (352) comprising: a plurality of rows, each row of the plurality of rows being associated with a corresponding unstructured data file (172) stored at the data repository (170); and a plurality of columns, each column of the plurality of columns comprising metadata (305) associated with the corresponding unstructured data file (172) for each row of the plurality of rows; and A structured data table (252) including the determined set of unstructured data files (172) is returned to the user (12).

12. The system (100) of claim 11, wherein the one or more unstructured data files (172) include image or audio data.

13. The system (100) of claim 11 or claim 12, wherein determining the set of unstructured data files (172) comprises, for each unstructured data file (172) of the object table (352): determining, based on the metadata (305) of the corresponding unstructured data file (172), that the corresponding unstructured data file (172) matches the query parameter (21); and Based on determining that the corresponding unstructured data file (172) matches the query parameter (21), the corresponding unstructured data file (172) is added to the collection of unstructured data files (172).

14. The system (100) of any one of claims 11 to 13, wherein the operations further include generating the object table (352) using a machine learning model before receiving the unstructured data query (20A) from the user (12).

15. The system (100) of claim 14, wherein generating the object table (352) comprises: selecting the one or more unstructured data files (172) for the object table (352) based on an object table query (20B); generating the plurality of rows based on a quantity of the one or more unstructured data files (172); determining the metadata (305) included in the object table (352) using the machine learning model; generating the plurality of columns based on the metadata (305); and The object table (352) is populated with references to the one or more unstructured data files (172) and the metadata (305) associated with each corresponding unstructured data file (172).

16. The system (100) of any one of claims 11 to 15, wherein the operations further comprise generating the object table (352) using a data crawler prior to receiving the unstructured data query (20A) from the user (12).

17. The system (100) of any one of claims 11 to 16, wherein the metadata (305) comprises at least one of a number of bytes, a file type, a creation time, a location, or business metadata (305).

18. The system (100) of any one of claims 11 to 17, wherein at least one row of the object table (352) is associated with a row access policy (310) that restricts access to the corresponding unstructured data file (172).

19. The system (100) of any one of claims 11 to 18, wherein the operations further comprise periodically updating the object table (352) based on changes at the data repository (170).

20. The system (100) of claim 19, wherein the object table (352) is updated based on a refresh rate (22).

Images