System and method for managing an artificial intelligence model

Abstract

A device may generate a source list identifying at least one source from a plurality of potential data sources. A device may queue the at least one sources for data acquisition based at least in part on last-interaction date. A device may scrape data from each of the at least one sources according to the queueing. A device may generate a relationship map of the scraped data yielding related data, and the related data is related to other scraped at or above the threshold. A device may output the related data as processed training data. A device may generate a large language model using the processed training data.

Description

SYSTEM AND METHOD FOR MANAGING AN ARTIFICIAL INTELLIGENCE MODELCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and benefits from United States Provisional Patent Application Serial No. 63 / 537,421, entitled “System and method for Managing an Al Model”, and filed on September 8, 2023. The aforementioned application is incorporated by reference herein as if set forth in its entirety.BACKGROUND

[0002] Large language models (LLMs) have become increasingly popular and utilized in various prompt / response applications such as search engines, customer service, etc. LLMs often include generative Al models trained based on extensive datasets built on historical or bulk general information. Thus, each instance of the LLM is only temporally relevant (or accurate) based on (1) the date of the training data used to train the LLM and / or (2) the date of the training of the LLM itself. For instance, it is widely understood that ChatGPT3 was trained on data up to September 2021. Thus, the ChatGPT3 LLM is limited in its ability to respond to prompts requesting information about data occurring after September 2021. Even if a training dataset is updated to include training data post September 2021, the LLM must be updated (e.g., retrained) before it can respond to prompts utilizing said updated training data.BRIEF DESCRIPTION OF THE FIGURES

[0003] Figure 1 depicts a system for managing an artificial intelligence (Al) model, in embodiments.

[0004] Figure 2 depicts a workflow for data acquisition and provisioning for use in managing an Al model, in embodiments.

[0005] Figure 3 depicts an example workflow diagram implemented by the system of Figure 1, in an embodiment.

[0006] Figure 4 shows an example data source, and the delineation of paragraphs therein.DETAILED DESCRIPTION OF THE EMBODIMENTS

[0007] The embodiments disclosed herein realize that LLM models become outdate when they are not trained on data that is current. The embodiments disclosed herein address this realization by implementing system(s) and method(s) for monitoring the training data used to train a given LLM, and updating said training data in real time. The embodiments also realize that updating (retraining) an LLM is costly both in time and money. The disclosure herein address this realization by implementing system(s) and method(s) that trigger model updates to the LLMs based on, in at least some embodiments, monitoring material used to provide the training data over time and initiating updates to the training data and / or LLMs when the training data changes beyond a threshold.

[0008] Figure 1 depicts a system 100 for managing an artificial intelligence (Al) model, in embodiments. System 100 includes model manager 102. Model manager 102 operates as an intermediate system between an end user device 104, data source(s) 106, and / or model source(s) 108. Model manager 102 may be hosted in a server external to end user device 104, data sources 106, and / or model sources 108. For example, model manager 102 may be hosted on one or more cloud-based operating system) such as Amazon Web Services (AWS)®, Microsoft Azure®, and the like. As another example, model manager 102 maybe hosted on a dedicated server not associated with a cloud-based operating system. In either the cloud and dedicated server embodiments Model manager 102 may be implemented on a single device, or implemented on multiple devices such as via a distributed processing system which may be parallel processing system, or serial processing system, or a combination thereof.

[0009] Model manager 102 may be configured under control of one or more administrator devices 130. Administrator device(s) 130 may include one or more input / output (I / O) devices 132. I / O device 132 may be a mouse / keyboard, touch screen, microphone / speaker combination, display, and any combination thereof. As such end user device(s) 130 may be one or more computer, tablet, laptop, smartphone, kiosk, or other device(s) capable of interacting with an administrator 135.

[0010] Model manager 102 may include one or more processing devices 110 in operational communication with one or more memory devices 112. Model manager 102 may include one or more wired and / or wireless-based communication interfaces 114 used to transmit data to, and received data from, one or more of end user device 104, data sources 106, and model sources 108 or other devices external to model manager 102. Each of the oneor more processing devices 110 may be a single processing device, or multiple processing devices, each having single or multiple cores. As such, it should be appreciated that the one or more processing devices 110 may be co-located or distributed based on the hardware / software / firmware configuration implementing said one or more processing devices 110.

[0011] The one or more memory devices 112 may store non-transitory computer readable instructions that, when executed by the one or more processing devices 110 operate to implement the functionality of model manager 102 as discussed herein.

[0012] A user 105 interacts with an input / output (I / O) device 116 of end user device 104 to query a large language model 118 managed by model manager 102. I / O device 116 may one or more of a mouse / keyboard, touch screen, microphone / speaker combination, display, and any combination thereof. As such end user device 104 may be a computer, tablet, laptop, smartphone, kiosk, or other device capable of interacting with user 105.

[0013] A key disadvantage of many current LLMs is the fact that they are temporally limited to knowledge of data used to train the LLM. Model manager 102 addresses this disadvantage by using one or more of data acquisition engine 120, data store 122, data manager 124, and model updater 126 to manage LLM 118 and keep it appropriately up-to- date based on the application thereof. Each of data acquisition engine 120, data manager 124, model updater 126 may be sub-instructions of the non-transitory computer readable instructions of model manager 102 that, when executed by the one or more processing devices 110 operate to implement the functionality of data acquisition engine 120, data manager 124, model updater 126, respectively, as discussed herein.DATA ACQUISITION

[0014] Data acquisition engine 120 may be implemented as stored non-transitory computer readable instructions that, when executed by the one or more processing devices 110 operate to implement the functionality of data acquisition engine 120 as discussed herein. For example, data acquisition engine 120 may implement a data crawling functionality that operates to scrape data from one or more of the data sources 106.

[0015] Data sources 106 may be publicly accessible websites, publication sources (e.g., newspapers, internet-based publishing websites, Elsivier, Springer, IEEE, or other industry and non-industry focused publishers), internal data stores (e.g., inventory management systems, Enterprise resource planning (ERP) data stores, customer relationship management (CRM) platforms, etc.), and other data sources. Data sources 106 may includethose that are capable of being “crawled” (e.g., scrubbed periodically or on an ad-hoc basis to obtain information therefrom). Data sources 106 may be stored as a master list and sub list. Examples of master-list data sources 106 include business-list websites (e.g., www.yellowpages.com, www.yelp.com) industry associations, self-help and enthusiast websites (e.g., www.bobvila.com), and aggregators (e.g., www.McMaster-Carr.com; www.Grabyar.com; www.walmart.com; www.amazon.com; etc.). The sub-lists may be generated from interactions with the master list sources resulting in a dynamic list of derivative sources (websites) that are associated with a given company or concept. As the sub-list or master list is updated, the available set of data sources 106 is updated and queued for additional data gathering via data acquisition engine 120 as discussed below.

[0016] Data sources 106 may be sorted using a given large language model. Business-listing sites, such as www.yellowpages.com act as a master list of businesses. Data sources 106 may include a listing of every organization in the given business-listing site. As data acquisition engine 120 is configured to focus on a given industry / application, data acquisition engine 120 may utilize the LLM to divide up all the available organizations listed in data sources 106 to filter for only a subset of the data sources 106 for use in retraining a given LLM for use by user 105. Moreover, as additional data sources 106 are identified, data acquisition engine 120 can flag the concept for addition to data sources 106 and retraining of a given LLM for use by user 105. Lor example, if a given LLM is trained on hardware store data, but needs to be updated to add restaurant data, data acquisition engine 120 is flagged with this concept and additional data sources 106 are obtained so that the given LLM is update d / retrained.

[0017] Data acquisition engine 120 may be configurable according to one or more configuration settings 134. Configuration settings 134 allow for customization of the types of data, and relevant data sources 106 that are monitored and scraped by data acquisition engine 120. One or more configuration settings 134 may be set via administrator 135 interacting with device(s) 130 via I / O device(s) 132. Lor example, if the model manager 102 is managing an LLM 118 used to provide replies to queries about retail data, then the one or more configuration settings 134 may be defined to indicate various types of websites associated with that retail data. As an example, if the retail data is hardware retail, then the one or more configuration settings 134 may identify various hardware store websites (e.g., www.homedepot.com; www.lowes.com; www.target.com;, etc.) to identify inventory and pricing information thereof. Moreover, one or more configuration settings 134 may identify one or more consumer product websites such as www.consumerreport.com ,www.bobvila.com, etc. to obtain review information about various products (which may or may not overlap with the products on the other data sources identified in the one or more configuration settings 134).

[0018] Acquisition orchestrator 136 operates according to the one or more configuration settings 134 to scrape data from the data sources 106. Data manager 124 may then operate to provision the data according to one or more LLM configurations 142.

[0019] Data scraped from data sources 106 is queued and stored as raw source data 138 in data store 122. Data manager 124 then provisions the raw source data 138 as processed data 140 into a standard format. Particularly, each data sources 106 may have a different format, naming convention, or other configuration of the data thereon. The data manager 124 provisioning the raw source data 138 allows the a model of each target data source 106 (e.g., website) and possible data subset in order to properly process it for actual use by the large language models. Data manager 124 operates to provision the raw source data 138 into processed data 140 according to a LLM configuration 142. LLM configuration 142 identifies the type and format of how processed data 140 is saved in the data store 122.

[0020] FIG. 2 depicts a workflow 200 for data acquisition and provisioning for use in managing an Al model, in embodiments. In one embodiment, all or some of workflow 200 is implemented upon execution of data acquisition engine 120 and / or data manager 124 of system 100.

[0021] At step 202, workflow 200 may include vetting and / or identify data sources to generate a source list 204 from the field of potential data sources 106. Step 202 may include administrator 135 interacting with device(s) 130 via I / O devices 132, to identify and queue ones of the data sources 106 for data scraping using data acquisition engine 120. Additionally or alternatively, step 202 may include querying a machine-trained model (such as a generated LLM; hereinafter “source -identifier model”) to identify potential ones of the data sources 106 that are relevant to a given application, technology field, etc. The output from the sourceidentifier LLM may be stored in source list 204.

[0022] At step 206, workflow 200 may include a source mapping process to generate a source map 208 of the sources identified in source list 204. Source map 208 may include information about when each source in source list 204 was last interacted with (e.g., scrapped, verified for accuracy, etc.). Source map 208 may also include queue identifying which sources 208 are to be scraped next. The queue may be prioritized based on lastinteraction date. If the given source is brand new, it may be prioritized for immediate data scraping over other previously-scraped sources.

[0023] Step 206 may be implemented periodically, or based on last interaction date of each source within source list 204. Reiterations of step 206 (indicated by step 230, discussed below) may result in updating current source map 208. If a given source fails (e.g., a website or subpage thereof is non-responsive), then the source may be dropped from the source map 208. In embodiments, the entire source may be dropped from source map 208 or in other embodiments only a portion of the source may be dropped from source map 208 so as to keep pertinent data from a given website and delete or ignore impertinent data from said given website. These embodiments, do not necessarily maintain canonical history of information within the generated LLM 118. Instead, they provide the advantage of providing current and relevant information via the LLM 118. If a product provided via a source from source list 204 is removed from the market (e.g., no longer available within any of data sources 106), then it is potentially of no use to the user 105. As such, by removing said source from source list 204, the LLM 118, once retrained, will only provide relevant information and not information about the non-provided product.

[0024] At step 210, workflow 200 operates to crawl and scrape data from each data sources 106 according to the queue in source map 208. The specific sites crawled may be identified based on the queue in source map 208, and / or source list 204 stored in data store 122. Source list 204 may be a component of one or more configuration settings 134. Step 210 may be implemented via data acquisition engine 120 and / or acquisition orchestrator 136 discussed above in Fig. 1.

[0025] In an embodiment of step 210, where a given data source in data sources 106 is a webpage, step 210 may include obtaining a sitemap link 212 forthe website that defines any link according to WWW standards, and then storing raw HTML data 214 from the webpage and any sub-link of the given webpage data source defined by the sitemap link 212. In some embodiments of step 210, the raw HTML data 214 stored does not include images and / or outer edges of the website document. For example, the outer edges include website formatting information, margin information, or other website design information.

[0026] The scraped data is stored as raw source data 138 as shown in FIG. 2. Initial instances of a given site crawl may result in unstructured data that has no relationship to any of the other information being pulled in. Accordingly, raw source data 138 may be a massive dataset. In order to reduce the dataset size, and yield data that is useable to system 100, step 216 includes generating a relationship map 218 of information within a given source, as well as information between disparate sources. For example, information about pricing of a specific product from www.homedepot.com may be related to review information about thatproduct, manufacturer, serial number range, etc. from www.bobvila.com. The relationship map 218 yields orphaned data 220 and related data 222. Each orphaned data 220 is referenced with orphaned data ID 224, which identifies the source within source list 204 that yielded the given orphaned data 220. Each related data 222 is referenced with related data ID 226, which identifies the source within source list 204 that yielded the given related data 222.

[0027] The related data 222 may include the raw source data 138 parsed into groups. For example, the parsing may include the raw source data 138 split out based on “paragraphs.” FIG. 4 shows an example data source, and the delineation of paragraphs therein. As discussed above raw source data 138 may not include images, so the picture of the dishwasher in paragraph 1 may not be included in the raw source data 138. Paragraphs may mean sub-portions of text data from the data sources 106. Each sub-portion may indicate a type of data (e.g., review, price data, etc.). In some embodiments, paragraphs are delineated when the text therein has a relationship, as defined by the relationship map 218 to text from other data sources such that the given paragraph may be correlated to another paragraph from another data source. In Figure 4, paragraph 1 indicates product type; paragraph 2 indicates product description; paragraph 3 indicates product features; and paragraph 4 indicates product components.

[0028] Orphaned data 220 is data that has no relationship to other types of data (or below a threshold number of relationships to other data). Data manager 124 may trim (e.g., filter, parse, delete, etc.) sources within source list 204 where a given source yields only orphaned data 220. This removal of sources from source list 204 may be permanent, or it may be only for a given period so that future iterations of data crawling is sped up while being able to periodically determine if a given source (or given task for the output LLM 118) has changed such that the removed source is now relevant and producing relevant data as opposed to orphaned data. Other types of filtering may occur with respect to source list 204. For example, say a given country becomes embargoed, one or more configuration settings 134 may be updated to ignore sources associated with that country. As such, the source list 204 may be updated to remove, or deactivate (e.g., “turn off’ or “ignore) those sources when data acquisition engine 120 is operating to crawl data sources 106.

[0029] Step 230 includes reiterating steps 204, 206, 210, 216, and 228 to periodically update that training data used to train LLM 118. This results in continued updating of the LLM 118 according to the needs of user 105. Importantly, unlike current implementations of LLMs, the reiteration may be performed periodically, or in response to changes in the training data used to previously train the LLM 118, or in response to information derivedfrom feedback 148. For example, in situations where the reiteration is implemented in response to feedback 148, the feedback may trigger changes to the source or type or use of data (e.g., changes in the relationship map 218 where data was incorrectly related, and / or addition of sources from data sources 106 to source list source list 204, and / or weighting of certain data, etc.) that resulted in poor replie s / outputs from the LLM 118. Continued training of the LLM 118 allows the LLM 118 to stay current, and thereby provide more accurate information to the user 105 utilizing said LLM 118.

[0030] The above discussion shows data being stored within a single instance of data store 122. It should be appreciated, however, that any piece of data may be stored in one or more datastores, or in separate and distinct datastores than other pieces of data. Moreover, each individual step of workflow 200 may be implemented by a single processing core or server, multiple processing cores or servers. Moreover, each individual step of workflow 200 may be implemented by the same server or servers, or distinct and separate server or servers from other of the steps. For example, source list generation step 202 may be implemented by a first server or servers that is the same or different from any one or more of source mapping step 206, data crawl / scrape step 210, relationship mapping step 216, and / or processed data output step 228. As another example, source mapping step 206 may be implemented by a first server or servers that is the same or different from any one or more of source list generation step 202, data crawl / scrape step 210, relationship mapping step 216, and / or processed data output step 228. As another example, data crawl / scrape step 210 may be implemented by a first server or servers that is the same or different from any one or more of source list generation step 202, source mapping step 206, relationship mapping step 216, and / or processed data output step 228. As another example, relationship mapping step 216 may be implemented by a first server or servers that is the same or different from any one or more of source list generation step 202, source mapping step 206, data crawl / scrape step 210, and / or processed data output step 228. As another example, processed data output step 228 may be implemented by a first server or servers that is the same or different from any one or more of source list generation step 202, source mapping step 206, data crawl / scrape step 210, and / or relationship mapping step 216.LLM GENERATION / RETRAINING

[0031] It should be appreciated that the term “train” as used herein may include generating or building a new LLM, or updating an existing LLM (particularly when the term “retrain” is used). When needed to generate or retrain LLM 118, model updater 126 may beimplemented to pull processed data 140 (output in step 228 in Fig. 2, above) to retrain LLM 118. If model updater 126 is a separate component (e.g., a separate server than the location of data acquisition engine 120, data store 122, and data manager 124), then processed data 140 maybe transmitted from data store 122 to another database associated with said separate server. Model updater 126 may implement an orchestration layer that divides up data for processing by dozens of servers and brings it back for completion into, or as, the final LLM 118.

[0032] Model updater 126 may be initiated periodically, or in response to the raw source data 138, and / or processed data 140 changing with respect to a previous version over a threshold amount, and / or feedback 148 indicating incorrect or poor responses from LLM 118, or for other reasons.

[0033] Model updater 126 may generate or retrain LLM 118 based on a new LLM protocol, or may utilize one or more externally-sourced LLMs 144. Externally-sourced LLMs 144 may be obtained from LLM source(s) 108. For example, externally-sourced LLMs 144 may include the GPT LLM (any version thereof) developed by OpenAI, the GPT LLM (any version thereof) developed by EleutherAI, Ernie 3.0 Titan developed by Baidu, megatron-Turing NLG developed by Microsoft and Nvidia, LaMDA LLM developed by Google, Galactica LLM developed by Meta, BloombergGPT developed by Bloomberg L.P., and any other publicly available, or otherwise licensed, LLM obtained from source(s) 108. In such instances of using one or more externally-sourced LLMs 144, model updater 126 may retrain the base one or more externally-sourced LLMs 144 using processed data 140. One or more externally-sourced LLMs 144 may be stored in data store 122 or on a separate database without departing from the scope hereof. Locating the one or more externally-sourced LLMs 144 in a separate database allows for scalability of the model manager 102 as the necessary data storage amount can be manipulated according to the desired one or more externally- sourced LLMs 144.

[0034] Moreover, client-specific data 146 may be combined with, or used instead of processed data 140. Client-specific data 146 encompasses “partner, client, external” data that is more static that allows for better training outcomes according to the desired application of LLM 118. Moreover, client-specific data 146 may include proprietary information that is not available from publicly available sources 106. This proprietary information could be holiday schedules, sales events, internal sales information, customer lists, pricing information, customer reviews, customer return data, and the like. Client-specific data 146 may include end-user data, such as flat files (comma separated or Excel-based files), direct APIintegration (e.g., a customer connects a database system and regularly pushes data to data acquisition engine 120 or data store 122), a webpage wizard for pasting data, proprietary CRM data (e.g., Salesforce data) or multiples (CVS / excel sheet options with more flexibility). Client-specific data 146 may further or alternatively include Enterprise Resource Management system information, this could be inventory data, product and SKU data, or other relatable items such as product descriptions, recall information, dimensions, manufacturer-suggested-retail-price (MSRP) by regionality, production numbers, safety data (MSDS or procedures like “don’t touch water heater when its turned on, allow to cool for 1 hour prior to maintenance or draining”), service manuals or assembly manuals (for example installation and / or assembly instructions), compatible replacement parts. Client-specific data 146 may further or alternatively include CRM data. A manufacturer generally knows all their wholesalers and retailers and their associated information. For example Google may not know everyone who will service a Kirby vacuum or a Samsung dryer but the Kirby CRM may have listing of service providers therefore. Client-specific data 146 may include data stored behind firewalls that require login via an usemame / password page, for example service manuals and support information often stored as PDFs.

[0035] The LLM 118 is then utilized by user 105. For example, device 116 may include an interactive user interface that allows the user 105 to query LLM 118 for information. End user device 104 may access an API that interfaces with the device storing LLM 118. The query submitted by user 105 is then processed by LLM 118 to generate a response.

[0036] The user 105 may additionally provide, via interaction with end user device 104, feedback 148. Feedback 148 may be stored in data store 122 (or any other database) and used by data manager 124, or administrator 135 via interaction with device(s) 130. Feedback 148 may be processed by data acquisition engine 120 to modify and / or generate one or more configuration settings 134. For example, feedback 148 may indicate that a given review is not relevant. The one or more configuration settings 134 may then be modified to remove, or reduce the weight of the data provided from, the source 106 providing said review. Additionally or alternatively, the feedback 148 may be used to modify the training data used by the source-identifier LLM implemented in step 202 discussed above with respect to FIG. 2. Additionally or alternatively, feedback 148 may be processed by data manager 124 to modify LLM configuration 142 . For example, feedback 148 may indicate that the provided response to request from user 105 was not relevant. Data manager 124 may then analyze the efficacy of LLM 118 and change the format or type of processed data 140 as saved in datastore 122. Additionally or alternatively, feedback 148 may be processed by model updater 126 to modify the one or more externally-sourced LLMs 144 or client-specific data 146 used to generate or modify LLM 118. For example, feedback 148 may indicate that the user 105 was requesting client-specific data, but the response provided by LLM 118 did not include any of the desired client-specific data. In response, the model updater 126 may retrain LLM 118 and include the desired client-specific data 146 missing based on the feedback 148.

[0037] FIG. 3 depicts an example workflow diagram 300 implemented by system 100, in an embodiment. Workflow diagram 300, system 100, and workflow 200 may be described in a retail industry application space (e.g., including product relation, product detail providing, inventory and consumer information production, etc.). However, it should be appreciated that the workflow diagram 300 may apply to additional types of application spaces, such as sales information, restaurant information providing (including finding restaurants and locations based on novel questions prompted to LLM 118 such as food issues, coffee type, diet restrictions, etc.), fashion information (e.g., fast fashion, trends, ecommerce data), services and service industries (e.g., roofing company (or other service) scheduling information, trade information, related trade information (e.g., where reviews of a company indicate they are good at a trade that is not advertised by that specific company), city and regional know-how (e.g., comparison between two or more cities for a specific type of vacation such as with or without kids), etc.

[0038] Block 302 implements a data acquisition and storage block 302. For example, data acquisition engine 120 may operate to scrape information from retail and consumer websites 304. Retail and consumer websites 304 are an example of data sources 106 and may include any publicly available website that provides pricing, inventory, review, consumer satisfaction, recall, or other product information such as but not limited to www.homedepot.com; www.lowes.com; www.target.com; www.consumerreport.com , www.bobvila.com, etc. Data acquisition engine 120 then stores the scraped data as raw source data 138 in data store 122.

[0039] Block 306 implements a data provisioning step. For example, data manager 124 may process raw source data 138 according to LLM configuration 142 and save the data as processed data 140 in data store 122. Blocks 302-306 may include and / or be replaced with workflow 200 discussed above.

[0040] Block 308 implements a step of gathering necessary information to train or retrain a base LLM. Block 308 may be implemented by data manager 124 and / or model updater 126 operating to obtain all or some of processed data 140 in order to train or retrainLLM 118. The obtained processed data 140 is shown in a different data store than the data store that the raw source data 138 and provisioned processed data 140 according to block 306 is stored in. It should be appreciated, however, that the data need not be passed on to a different data store.

[0041] Block 308 may be triggered when a current version of the processed data 140 changes from a prior version of the processed data 140. For example, inventory changes in one or more of the retail and consumer websites 304 may change over time. The responses of the LLM 118 described above may be outdated and incorrect if the inventory is incorrect. System 100, and workflow 300, operate to automatically update LLM 118 (and trigger block 308 and the following blocks of workflow 300) in response to this change in the retail and consumer websites 304 so that the LLM 118 trained thereon is always current and providing accurate answers.

[0042] Block 310 operates to train or retrain a base LLM for deployment and use by a client. Block 310 may utilize one or more externally-sourced LLMs 144 from source(s) 108, or train / retrain a new LLM model. Moreover, block 310 may utilize the processed data 140, which is publicly available data, as well as client-specific data 146 which may be proprietary data or otherwise data specific to the user 105 that will be querying the deployed LLM.

[0043] Block 312 operates to deploy the trained / retrained LLM 118 to the user 105. Block 312 may include generating a user-interface that is displayed to user 105 via end user device 104. Block 312 may include receiving, via user 105 interacting with end user device 104 using I / O device 116, a request. Block 312 may utilize an API to transmit the request to the deployed LLM 118, allow the LLM 118 to process the request, and then transmit back a response to the end user device 104 for interpretation by user 105.

[0044] Block 314 gathers feedback from user 105, using end user device 104. For example. Feedback 148 is gathered from user 105, via user 105 interacting with end user device 104 using I / O device 116, based on the quality / accuracy perceived by user 105. Block 314 may further include modifying one or more further iterations of blocks 302 and / or 306 based on the feedback 148.

[0045] Changes may be made in the above methods and systems without departing from the scope hereof. It should thus be noted that the matter contained in the above description or shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. The following claims are intended to cover all generic and specific features described herein, as well as all statements of the scope of the present method and system, which, as a matter of language, might be said to fall therebetween.

Claims

CLAIMSWhat is claimed is:

1. A method for managing an Al model, comprising: generating a source list identifying at least one source from a plurality of potential data sources; queuing the at least one sources for data acquisition based at least in part on lastinteraction date; scraping data from each of the at least one sources according to the queuing; generating a relationship map of the scraped data yielding related data, and the related data is related to other scraped at or above a threshold; outputting the related data as processed training data; and generating a large language model using the processed training data.

2. The method of claim 1, wherein at least one of the sources is a website.

3. The method of claim 2, wherein the scrape data from the website includes: obtaining a sitemap link from the website; and pulling raw HTML data from the website and sub-links therein according to the sitemap link.

4. The method of claim 3, wherein the scrape data from the website includes ignoring images within the website.

5. The method of claim 1, further comprising updating a source map after the generating the large language model; reiterating queuing at least one source from the plurality of potential data sources for data acquisition based on the source map;reiterating the queuing, scraping data, generating a relationship map, and outputting the processed data for at least one source of the plurality of potential data sources according to the updated source map; and, updating the large language model based on processed training data yielded from the reiteration of outputting the processed training data.

6. The method of claim 5, wherein the updating the source map includes removing one or more of the at least one source used to generate a first iteration of the processed training data.

7. The method of claim 1, wherein generating the relationship map includes parsing the scraped data into sub-groups; and identifying relationship between text within sub-groups from a first data source to text within a different sub-group corresponding to a second data source.

8. The method of claim 7, wherein parsing the scraped data into sub-groups includes delineating each sub-group according to paragraphs of text.

9. The method of claim 8, wherein parsing the scraped data into sub-groups includes delineating each sub-group according to type of data therein.

10. The method of claim 9, the type of data including one or both of price data or review data.

11. The method of claim 1, the generating a relationship map of the scraped data yielding orphaned data; wherein the orphaned data includes scraped data that is related to less than a threshold amount of other scraped data.

12. The method of claim 11, further comprising removing sources associated with the orphaned data from the source list.

13. A system for managing an Al model, comprising: a processor; memory;a data acquisition and data manager module, comprising computer readable instructions stored in memory that, when executed by the processor, cause the system to: generate a source list identifying at least one source from a plurality of potential data sources, queue the at least one sources for data acquisition based at least in part on lastinteraction date, scrape data from each of the at least one sources according to the queueing, generate a relationship map of the scraped data yielding related data, and the related data is related to other scraped at or above a threshold; output the related data as processed training data, and generate a large language model using the processed training data.

14. The system of claim 13, wherein at least one of the sources is a website.

15. The system of claim 14, wherein the scrape data from the website includes: obtain a sitemap link from the website; and pull raw HTML data from the website and sub-links therein according to the sitemap link.

16. The system of claim 14, wherein the scrape data from the website includes ignoring images within the website.

17. The system of claim 14, comprising further computer readable instructions that, when executed by the processor, further cause the system to: update a source map after the generating the large language model; reiterate queuing at least one source from the plurality of potential data sources for data acquisition based on the source map;reiterate the queuing, scraping data, generating a relationship map, and outputting the processed data for at least one source of the plurality of potential data sources according to the updated source map; and, update the large language model based on processed training data yielded from the reiteration of outputting the processed training data.

18. The system of claim 17, wherein the updating the source map includes removing one or more of the at least one source used to generate a first iteration of the processed training data.

19. The system of claim 13, wherein generate the relationship map includes parsing the scraped data into sub-groups; and identifying relationship between text within sub-groups from a first data source to text within a different sub-group corresponding to a second data source.

20. The system of claim 19, wherein parsing the scraped data into sub-groups includes delineating each sub-group according to paragraphs of text.

21. The system of claim 20, wherein parsing the scraped data into sub-groups includes delineating each sub-group according to type of data therein.

22. The system of claim 21, the type of data including one or both of price data or review data.

23. The system of claim 13, the generate a relationship map of the scraped data yielding orphaned data; wherein the orphaned data includes scraped data that is related to less than a threshold amount of other scraped data.

24. The system of claim 23, comprising further computer readable instructions that, when executed by the processor, further cause the system to: remove sources associated with the orphaned data from the source list.

Images