COMPUTER-BASED SYSTEMS, COMPUTER COMPONENTS, AND COMPUTER OBJECTS CONFIGURED TO IMPLEMENT DYNAMIC OUTLIER BIAS REDUCTION IN MACHINE LEARNING MODELS - Patent application

Abstract

To provide a system and method for a dynamic outlier bias reduction machine learning model.SOLUTION: A method for determining a set of model parameters of a machine learning model includes: applying the machine learning model to training data; generating model prediction errors; generating a data selection vector to identify non-outlier target variables based on the model prediction errors; utilizing the data selection vector to generate a non-outlier data set; determining updated model parameters based on the non-outlier data set; repeating the above steps until a censoring performance termination criterion is satisfied; training classifier model parameters for an outlier classifier machine learning model; applying the outlier classifier machine learning model to activity-related data to determine non-outlier activity-related data; and applying the machine learning model to the non-outlier activity-related data to predict future activity-related attributes for the user activity.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure generally relates to improvements configured to implement bias reduction in machine learning models. Computer-based systems, computer components, and computer operating systems Regarding the object.

[0002] (Priority Claim) This application is a joint venture of “COMPUTER-BASED SYNTHETIC SYSTEMS” filed on September 18, 2019. STEMS,COMPUTING COMPONENTS AND COMPUTING OBJECTS CONFIGURED TO IMPLEMENT DYNAMIC OUTLIER BIAS REDUCTION IN MACHINE LEARN Priority to U.S. Provisional Application No. 62 / 902,074 entitled "ING MODELS" No. 6,313,635, filed on Oct. 23, 2006, and which is hereby incorporated by reference in its entirety.

[0003] (Copyright Notice) A portion of the disclosure of this patent document contains material that is subject to copyright protection. All Rights Reserved. The person who filed the application must be the person who filed the application. There is no objection to facsimile reproduction by any of the above references, but otherwise all Copyright reserved. The following notice applies to the drawings forming part of this document and the software described below. All rights reserved. er Inspection and Insurance Company, All Rights Reserved. Reproduction prohibited [Background technology]

[0004] Machine learning models make decisions based on patterns and inferences learned from sample / training data. The device may include one or more computers or processing devices for forming predictions or decisions based on the data. Bias in sample / training data selection is reflected in machine learning model predictions and decisions It can be done. Summary of the Invention

[0005] Embodiments of the present disclosure include a method for dynamic outlier bias reduction machine learning models. At least one user activity is detected by at least one processor. receiving a training data set of objective variables representing at least one activity-related attribute; and using the at least one processor to determine one or more outliers. receiving at least one bias criterion to be used for the at least one processor; determining a set of model parameters for the machine learning model by the (1) generating, by the at least one processor, a set of initial model parameters; Applying the machine learning model to the training data set to obtain a model predicted value (2) determining, by said at least one processor, a set of said models; By comparing a set of predicted values ​​with the corresponding actual values ​​in the training data set, (3) generating an error set of data element errors by said at least one processor; A bias criterion for the at least one bias criterion and the error set of data element errors is calculated by the Generate a data selection vector for identifying non-outlier target variables based, at least in part, on (4) by said at least one processor, Utilizing the data selection vector for each set of data to generate a non-outlier data set. (5) calculating, by the at least one processor, a set of non-outlier data based on the non-outlier data set; determining a set of updated model parameters for the machine learning model using the and (6) detecting, by said at least one processor, at least one aborted execution end. The previous step is repeated by repeating steps (1) to (5) as an iteration until the completion criterion is met. The model parameters for the machine learning model are whereby each iteration obtains a set of the updated model parameters. a set of predicted values ​​using the set of predicted values ​​as the set of initial model parameters; , regenerating the error set, the data selection vector, and the non-outlier data set. and, by said at least one processor, an outlier classifier based at least in part on the data set and the data selection vector; Training a set of classifier model parameters of a machine learning model. a trained outlier configured to identify at least one outlier data element; obtaining a threshold classifier machine learning model; Then, the trained outlier classifier machine learning model is applied to the at least one user. (i) applying the activity-related data to a dataset of activity-related data relating to user activities; a set of outlier activity-related data in the dataset; and ii) determining a set of non-outlier activity related data in the dataset; and computing the non-outlier activity related data components using the machine learning model by at least one processor. applying the disjoint set to generate future activity-related attributes related to the at least one user activity; and predicting the

[0006] Embodiments of the present disclosure include a system for dynamic outlier bias reduction machine learning models. The system is in communication with a non-transitory computer readable storage medium having software instructions stored thereon. and at least one processor that, when executed, at least one processor, receiving a training data set of a response variable representing another activity-related attribute; A sequence that receives at least one bias criterion that is used to determine one or more outliers. and determining a set of model parameters for the machine learning model, 1) Computing the machine learning model having a set of initial model parameters onto the training data. (2) applying the model predictions to a set of data to determine a set of model predictions; The data set is compared to the corresponding actual values ​​in the training data set. (3) generating an error set of data element errors; and Identifying non-outlier response variables based at least in part on the at least one bias criterion. (4) generating a data selection vector for the training data set; (5) generating a non-outlier data set using the data selection vector; Updated model parameters for the machine learning model based on the non-outlier data set. (6) determining a set of parameters for which at least one abort run termination criterion is met; The updated model is then iteratively updated by repeating steps (1) to (5) until the updated model is and obtaining a set of model parameters for the machine learning model as model parameters. whereby each iteration updates the set of updated model parameters by The set of predicted values, the set of errors, and regenerating the data set, the data selection vector, and the non-outlier data set. and at least partially extracting the training data set and the data selection vector. Based on the objective, we train a set of classifier model parameters of the outlier classifier machine learning model. and scanning the data in a manner configured to identify at least one outlier data element. obtaining a trained outlier classifier machine learning model; a trained outlier classifier machine learning model for the at least one user activity. (i) applying the method to a dataset of activity-related data, and ii) a set of outlier activity-related data in said dataset of activity-related data. determining a set of non-outlier activity related data in the a set of non-outlier activity related data elements to determine whether the at least one user activity is outlier-related; Predicting relevant future activity-related attributes.

[0007] The system and method of the disclosed embodiment further comprises, by the at least one processor: Applying the data selection vector to the training data set to perform outlier training. determining a sampling data set; and, by said at least one processor, Use the outlier training dataset to develop at least one outlier-specific machine learning model. Train at least one outlier-specific model parameter for each outlier data value and predicting, by the at least one processor, the outlier specific machine The learning model is utilized to generate outlier activity related data for the set of outlier activity related data. and predicting a value of the parameter.

[0008] The system and method of the disclosed embodiment further comprises, by the at least one processor: The training dataset is used to generalize the generalized machine learning model. training model parameters to predict data values; A processor is used to perform the outlier activity analysis using the generalized machine learning model. predicting outlier activity-related data values ​​for a set of activity-related data; and performing, by at least one processor, the generalized machine learning model to generate the activity. and predicting a dynamic associated data value.

[0009] The systems and methods of the disclosed embodiments include applying, by the at least one processor, the data selection vector to the training dataset to determine an outlier training dataset; training, by the at least one processor, outlier-specific model parameters of an outlier-specific machine learning model using the outlier training dataset to predict outlier data values; training, by the at least one processor, generalized model parameters of a generalized machine learning model using the training dataset to predict data values; utilizing, by the at least one processor, the outlier-specific machine learning model to predict outlier activity-related data values ​​for the set of outlier activity-related data; Generalized and utilizing a machine learning model to predict the activity-related data value.

[0010] The system and method of the disclosed embodiment further comprises, by the at least one processor: The training dataset is used to generalize the generalized machine learning model. training model parameters to predict data values; and generating, by a processor, the activity-related predicting said activity-related data value for a set of data; The outlier classifier machine learning model is utilized by a processor to generate the activity-related data. identifying an outlier activity associated data value of the data value; and removing, by a filter, the outlier activity related data values.

[0011] In the system and method of the disclosed embodiment, the training data set is A study of concrete compressive strength as a function of concrete composition and concrete hardening development. Contains at least one activity-related attribute.

[0012] In the system and method of the disclosed embodiment, the training data set includes said at least one activity of energy usage data as a function of yard environmental conditions and lighting conditions. Contains related attributes.

[0013] The system and method of the disclosed embodiment further comprises, by the at least one processor: An API (application program) for generating a forecast having at least one data element receiving a programming interface request; and a processor for scheduling execution of the machine learning model. instantiating a cloud computing resource; and Utilizing the machine learning model according to the schedule executed by a processor. and at least one activity-related data element value for said at least one data element. and predicting, by the at least one processor, a step of returning activity-related data element values ​​to a computing device associated with said API request; and

[0014] In the system and method of the disclosed embodiment, the training data set is the at least one activity-related attribute of the three-dimensional patient image of the medical data set; The learning model comprises: The activity-related data value may include a parameter.

[0015] In the systems and methods of the disclosed embodiments, the training data set is The at least one action-related attribute of the simulated control result for the child machine command. and the machine learning model is adapted to determine the activity function including a control command for the electronic machine. The method is configured to predict consecutive data values.

[0016] The system and method of the disclosed embodiment further comprises, by the at least one processor: dividing the set of activity-related data into a plurality of subsets of activity-related data; a plurality of subsets of the activity-related data, Determining an ensemble model for each subset of the activity-related data The machine learning model includes an ensemble of models, and each ensemble model The model is a sequence that includes a random combination of models from the ensemble of models. and, by the at least one processor, using each ensemble model. , predicting ensemble-specific activity-related data values ​​individually; and by a processor of the ensemble specific activity related data value and a known value. determining an error in each ensemble model; and The processor selects the best performing ensemble model based on the minimum error. and [Brief description of the drawings]

[0017] Various embodiments of the present disclosure may be further described with reference to the accompanying drawings, in which: The drawings are not necessarily drawn to scale. Instead, emphasis is placed on generally illustrating the principles of the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting. The present invention is not intended to be limiting, but merely to teach one of ordinary skill in the art various uses of one or more exemplary embodiments. should be interpreted as a representative basis for

[0018] [Figure 1] 1 depicts certain computer-based architectures illustrating certain example aspects of at least some embodiments of the present disclosure. [Diagram 2] 1 depicts certain computer-based architectures illustrating certain example aspects of at least some embodiments of the present disclosure. [Diagram 3] 1 depicts certain computer-based architectures illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 4] 1 depicts certain computer-based architectures illustrating certain example aspects of at least some embodiments of the present disclosure. [Diagram 5] 1 depicts certain computer-based architectures illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 6] 1 depicts a schematic flow chart illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 7] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 8] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 9] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 10] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 11] 1 depicts a schematic flow chart illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 12] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 13] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 14A] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 14B] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. [Figure 15] 1 shows graphs illustrating certain example aspects of at least some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Various detailed embodiments of the present disclosure are disclosed herein in conjunction with the accompanying drawings. However, it should be understood that the disclosed embodiments are merely exemplary. Each of the examples provided in connection with various embodiments of the present disclosure are illustrative and not limiting.

[0020] Throughout this specification, the following terms have the meanings expressly defined herein, unless the context indicates otherwise: As used herein, the terms "in one embodiment" and "in some embodiments" have meanings related to the The phrase "in several embodiments" does not necessarily refer to the same embodiment(s). In addition, as used herein, "in another embodiment" means that The phrases "in some other embodiments" and "in some other embodiments" do not necessarily refer to different embodiments. This is not necessarily the case, but it can be. Therefore, as explained below, The embodiments may be readily combined without departing from the scope or spirit of the present disclosure. do.

[0021] Moreover, the term "based on" is not exclusive, unless the context clearly indicates otherwise. In addition, throughout this specification, the term "indefinite coronary artery disease" refers to a disease that is not caused by a coronary artery disease. The meanings of the words "a", "an" and "the" include plural references. The meaning of "in" includes "inside" and "on" .

[0022] At least one aspect / functionality of the various embodiments described herein may be implemented in real time. It is understood that the implementation may be performed systematically and / or dynamically. The term "real-time" refers to the instantaneous or approximate occurrence of another event / action. Refers to events / actions that can occur instantaneously. For example, "real-time processing" and "relay "Real-time computation" and "real-time execution" all refer to the physical processes involved (e.g. The actual time that a user interacts with an application on a mobile device occurs It is related to the performance of calculations during the calculation, and the results of the calculations can be used to guide physical processes. do.

[0023] As used herein, the terms "dynamically" and "automatically" and their respective Logical and / or linguistic associations and / or derivations may be This means that an event can be triggered and / or occur without any human intervention. In some embodiments, events and / or actions according to the present disclosure are generated in real time. and / or nanoseconds, nanoseconds, milliseconds, milliseconds, seconds, seconds, minutes, minutes, hours , daily, several days, weekly, monthly, etc.

[0024] In some embodiments, an exemplary novel specially programmed The computer system operates in a distributed network environment and communicates with one or more suitable They communicate with each other via communication networks (e.g., the Internet, satellite, etc.) and are not limited to However, IPX / SPX, X.25, AX.25, AppleTalk(TM), TCP / IP (e.g. HTTP), Near Field Communication (NFC), RFID, Narrowband Internet of Things NBIOT, 3G, 4G, 5G, GSM, GPRS, WiFi , WiMax, CDMA, satellite, ZigBee, and other suitable communication modes The device may be configured to utilize one or more suitable data communication protocols / modes such as: In some embodiments, NFC is used by an NFC-enabled device to communicate with a “swipe,” “bump,” or “tap” " or may refer to a short-range wireless communication technology that is otherwise brought into close proximity for communication. .

[0025] The material disclosed herein may be implemented as software or firmware, or any combination thereof. or machine readable and executable by one or more processors. The program may be implemented as instructions stored on a machine-readable medium. Any medium for storing or transmitting information in a form readable by a computer device, For example, the machine-readable medium may include a read-only memory (ROM), a run-time memory (RAM), Random Access Memory (RAM), magnetic disk storage media, optical storage media, flash memory Devices, electrical, optical, acoustic or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.

[0026] As used herein, the terms "computer engine" and "engine" , other software and / or hardware components (e.g. libraries, software Software Development Kit (SDK), objects, etc.) At least one software component and / or at least one A combination of one software component and at least one hardware component. Shows a combination.

[0027] Examples of hardware elements include processors, microprocessors, circuits, circuit elements ( (e.g. transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific Application specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors ( DSP), Field Programmable Gate Array (FPGA), Logic Gates, Registers , semiconductor devices, chips, microchips, chipsets, etc. In some embodiments, one or more of the processors may be a complex instruction set computer (CISC) or reduced instruction set (RE) processor. RISC (Reduced Instruction Set Computing) processor, a processor compatible with the x86 instruction set processor, multi-core, or other microprocessor or central processing unit (CPU) In various implementations, one or more processors may be implemented as dual The core may be a single-core processor, a dual-core mobile processor (multiple).

[0028] Examples of software include software components, programs, and applications. , computer programs, application programs, system programs, machines Programs, operating system software, middleware, firmware, Software module, routine, subroutine, function, method, procedure, software Software Interface, Application Program Interface (API) , instruction set, computer code, computer code, code segment, computer The information may include a data code segment, a word, a value, a symbol, or any combination thereof. Whether the embodiment is implemented using hardware and / or software elements Determining the desired computation speed, power level, thermal tolerance, processing cycle budget, input input data rate, output data rate, memory resources, data bus speed, and other design or It may vary according to any number of factors, such as performance constraints.

[0029] One or more aspects of at least one embodiment represent various logic within a processor. The implementation may be represented by instructions stored on a machine-readable medium, which may be executed by a machine. When read, the code causes a machine to create logic to perform the techniques described herein. Such expressions, known as "IP cores", are used to create logic or processors. The software is stored on a tangible machine-readable medium and distributed to various customers or manufacturing facilities for loading into the manufacturing machines. It should be noted that the various embodiments described herein may, of course, be However, any suitable hardware and / or computer software language (e.g., C++, Objective-C, Swift, Java, JavaScript, Py It may be implemented using java.thon, Perl, QT, etc.

[0030] In some embodiments, one or more of the exemplary novel computer-based devices of this disclosure At least one personal computer (PC), laptop computer, Ultra laptop computers, tablets, touchpads, portable computers , handheld computers, palmtop computers, personal digital assistants (PDA), cellular phone, combined cellular phone / PDA, TV, smart device (smart Smartphones, smart tablets, smart TVs, etc.), mobile internet devices including, in whole or in part, a mobile integrated device (MID), messaging device, data communication device, etc. may be incorporated therein.

[0031] As used herein, the term "server" refers to a processing, database, and communication This should be understood to refer to a service point that provides functionality, but is not limited to this. By way of example, the term "server" includes associated communication and data storage devices and databases. It may refer to a single physical processor having computing capabilities, or it may refer to a processor and associated network Networked or clustered complexes of networks and storage devices, and servers the operating software that supports the services provided by It may also refer to database systems and application software. Ba is an example.

[0032] In some embodiments, exemplary novel One or more of the computer-based systems may include files, contacts, tasks, and email. In any suitable form, such as a tweet, a map, an entire application (e.g., a calculator), etc. Any possible digital object and / or data unit (e.g., a particular application) Acquire, manipulate, transmit, store, transform, generate, and / or use the information (within and / or outside the application) In some embodiments, the present disclosure may provide a One or more of the exemplary novel computer-based systems include, but are not limited to, For example, (1) AmigaOS, AmigaOS4, (2) FreeBSD, NetBSD , OpenBSD, (3) Linux, (4) Microsoft Windows, ( 5) OpenVMS, (6) OS X (Mac OS), (7) OS / 2, (8) Sol aris, (9)Tru 64 UNIX(registered trademark), UNIX, (10)VM,(1 1)Android,(12)Bada,(13)BlackBerry OS,(14) )Firefox OS,(15)iOS,(16)Embedded Linux,( 17)Palm OS,(18)Symbian,(19)Tizen,(20)Web OS,(21)Windows Mobile,(22)Windows Phone, (23)Adobe AIR,(24)Adobe Flash,(25)Adobe Shockwave,(26)Binary Runtime Environment for Wireless (BREW), (27) Cocoa (API), (28) C ocoa Touch,(29)Java Platforms,(30)JavaFX ,(31)JavaFX Mobile,(32)Microsoft XNA,(33) )Mono,(34)Mozilla Prism,XUL and XULRunne r,(35).NET Framework,(36)Silverlight,(37) )Open Web Platform, (38)Oracle Database, ( 39) Qt, (40) SAP NetWeaver, (41) Smartface, (4 2) Vexi, and (43) Windows Runtime, etc. It may be implemented across one or more of the forms.

[0033] In some embodiments, an exemplary novel computer-based system of the present disclosure; and and / or an exemplary novel computer-based apparatus that implements features consistent with the principles of the present disclosure. Hardwired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, implementations consistent with the principles of the present disclosure may be configured for use with hardware The present invention is not limited to any specific combination of hardware circuitry and software. For example, various embodiments are , including but not limited to, stand-alone software packages, software packages The present invention may be embodied in many different ways as software components, such as Software parts that are individually designed or incorporated as "tools" into a larger software product. It may be a package.

[0034] For example, exemplary software specifically programmed in accordance with one or more principles of the present disclosure may be The software may be used as a standalone product or installed into an existing software application. It can be downloaded from a network, for example, a website, as an add-in package for installing the software. For example, the software may be specifically programmed in accordance with one or more principles of the present disclosure. The exemplary software implemented is a client-server software application. It may also be available as a web-enabled software application. For example, an exemplary software program specifically programmed in accordance with one or more principles of the present disclosure may include: It may be embodied as a software package that is installed on a hardware device. stomach.

[0035] In some embodiments, the exemplary novel computer-based system / platform of the present disclosure Exemplary novel computer-based devices, and / or exemplary novel computer-based systems. The computer-based component may be used by, but is not limited to, at least 100 users (e.g., At least 1,000 people (for example, but not limited to, 1 ,000 to 9,999), at least 10,000 (for example, but not limited to, 10, 000 to 99,999), at least 100,000 (for example, but not limited to, 10 0,000 to 999,999), at least 1,000,000 (e.g., not limited However, 1,000 to 9,999,999) and at least 10,000,000 people (e.g. , but not limited to, 10,000,000 to 99,999,999), at least 100 ,000,000 people (for example, but not limited to, 100,000,000 to 999,99 9,999), at least 1,000,000,000 people (for example, but not limited to, 1 ,000,000,000,000,000 to 10,000,000,000 The system may be configured to handle a large number of simultaneous users.

[0036] In some embodiments, the exemplary novel computer-based system and / or Alternatively, an exemplary novel computer-based apparatus may be implemented using a separate, specially programmed Implementing graphical user interfaces (e.g. desktop, web apps, etc.) In various implementations of the present disclosure, the final output may be configured to include, but is not limited to, However, the display may be a computer screen, a mobile device screen, etc. In various implementations, the display may be a holographic display. In various implementations, the display may be capable of receiving visual projection. Such projections may be transparent surfaces that display various forms of information, images, and / or audio. For example, such projections can be used in mobile augmented reality (MAR) applications. It can be a visual overlay for a variety of applications.

[0037] In this specification, the term "cloud," "internet cloud," "cloud computer," ", "Cloud Architecture" and similar terms mean: (1) a real-time communication network; (2) a large number of computers connected via a network (e.g., the Internet); Simultaneous programming on multiple computers (e.g., physical machines, virtual machines (VMs)) (3) providing the ability to run network-based programs or applications; A service that appears to be delivered by actual server hardware, but is actually It is provided by virtual hardware (e.g., a virtual server) and runs on one or more real machines. Simulated by software running on a network (e.g., impact on end users) Move and scale up (or down) in place without impact (enabling

[0038] In some embodiments, the exemplary novel computer-based system and / or Alternatively, the exemplary novel computer-based device may incorporate one or more encryption techniques (e.g., secret / Public key pair, 3DES (Triple Data Encryption Standard) rd), block cipher algorithms (e.g., IDEA, RC2, RC5, CAST, S kipjack), cryptographic hash algorithms (e.g., MD5, RIPEMD-160 , RTR0, SHA-1, SHA-2, Tiger(TTH), WHIRLPOOL, R NG) to store and / or transmit data securely. This is also fine.

[0039] The above examples are, of course, illustrative and not limiting.

[0040] As used herein, the term "user" means at least one user. In some embodiments, the terms "user," "subscriber," "consumer," or "customer" may be used. The term refers to a user and / or data processor of one or more applications described herein. This should be understood to refer to consumers of data provided by providers, without limitation. For example, the term "user" or "subscriber" refers to a user who is accessing the - Anyone who receives data via the Internet or data provided by a service provider or automated software that receives the data and stores or processes the data. This may also refer to an application.

[0041] FIG. 1 illustrates an exemplary method for reducing bias in machine learning in accordance with one or more embodiments of the present disclosure. 1 shows a block diagram of a typical computer-based system 100. Not all components are required to implement one or more embodiments. Changes in the arrangement and type of components may depart from the spirit or scope of the various embodiments of the present disclosure. In some embodiments, the exemplary computer-based system An exemplary novel computer device and / or an exemplary novel computer computer of the system 100 A component may have multiple members and / or concurrent transactions, as described in more detail herein. In some embodiments, an exemplary computer The database system / platform 100, as described in the embodiments herein, In addition, dynamic outlier bias reduction (DOBR) evaluation, caching, retrieval, and / or database access of data, including Scalable computer and / or network systems incorporating various strategies for continuous pooling An example of a scalable architecture is based on a multi-network architecture. This is an architecture capable of running multiple servers.

[0042] In some embodiments, referring to FIG. 1, an exemplary computer-based system 1 Members 102-104 (e.g., clients) of the network 100 are connected to a network such as a network 105. For example, the servers 106 and 107 Virtually any computer that can send and receive messages to and from other computer devices, such as In some embodiments, the member devices 102-104 may include a computer device. Personal computers, multiprocessor systems, microprocessor-based or It may be a programmable consumer electronic device, a network PC, etc. In this embodiment, one or more of the member devices 102 to 104 is typically a mobile phone, a smartphone, or the like. Smartphones, pagers, walkie-talkies, radio frequency (RF) devices, infrared (IR) devices A mobile device, a CB, an integrated device combining one or more of the above devices, or virtually any mobile This may include a computer device that is connected using a wireless communication medium, such as a computer device. In some embodiments, one or more of the member devices 102-104 may be a PDA. , Pocket PC, Wearable Computer, Laptop, Tablet, Desktop Computers, netbooks, video game devices, pagers, smartphones, ultra Mobile personal computers (UMPCs) and / or wired and / or wireless communication media (For example, NFC, RFID, NBIOT, 3G, 4G, 5G, GSM, G PRS, WiFi, WiMax, CDMA, satellite, ZigBee, etc. Any other device that is set up to communicate with the In some embodiments, one of the member devices 102-104 may be a device capable of The member devices include, among other things, internet browsers, mobile applications, audio One or more applications such as voice calling, video games, video conferencing, and email. In some embodiments, one of the member devices 102-104 may include and execute the above. One or more member devices may be configured to receive and transmit web pages, etc. In some embodiments, the exemplary specially programmed browser application of the present disclosure The application may be a SMGL such as, but not limited to, HyperText Markup Language (HTML). (Standard Generalized Markup Language), None Handheld communication such as Wireline Application Protocol (WAP) and Wireless Markup Language (WML) Held Device Markup Language (HDML), WMLScript, XML, Java You can use virtually any web-based language, including Script, to create graphics, text, and The device may be configured to receive and display text, multimedia, etc. In this embodiment, the member devices 102 to 104 are implemented using Java, .Net, QT, C , specially programmed in either C++ and / or other suitable programming languages In some embodiments, one or more of the member devices 102-104 may The device may perform functions such as, but not limited to, messaging, browsing, searching, playback, streaming, etc. or locally stored or uploaded messages, images and / or videos, and / or perform a variety of possible tasks, such as displaying various types of content, including games. For this purpose, the device may be specially programmed to contain or execute applications.

[0043] In some embodiments, the exemplary network 105 may include any Providing network access, data transfer and / or other services to computer devices In some embodiments, the exemplary network 105 may be, for example, a limited However, GSM (registered trademark) (Global System for Mobile communications) communication Association, IETF (Internet Engineering Task Force) Networking Task Force), and WiMAX (Worldwide Interconnection (Interoperability for Microwave Access) Forum At least one standard that may be based, at least in part, on one or more standards established by the A special network architecture may be included and implemented. The exemplary network 105 may be based on a GSM architecture, a GPRS (Ge Universal Packet Radio Service (UMTS) architecture Universal Mobile Telecommunications System em) architecture, and LTE (Long Term Evolution) In some embodiments, the UMTS may implement one or more of the following evolutions: Alternatively or in combination with one or more of the above, the network 105 may include a Wi It may include and implement the WiMAX architecture defined by the MAX Forum. In some embodiments, and optionally, any of the embodiments described above or below. In combination, the exemplary network 105 may be, for example, a local area network (LON). Network (LAN), Wide Area Network (WAN), Internet, Virtual LAN (VLAN), Corporate LAN, Layer 3 Virtual Private Network (VPN), Corporate I P network, or any combination thereof. In some embodiments, and optionally, any of the embodiments described above or below. In combination, at least one computer network on the exemplary network 105 Network communications include, but are not limited to, NFC, RFID, and narrowband Internet of Things (NBI). NBIOT), ZigBee (registered trademark), 3G, 4G, 5G, GSM (registered trademark), G A few of the above include PRS, WiFi, WiMax, CDMA, satellite, and any combination thereof. In some embodiments, the cellular phone may be transmitted based at least in part on one communication mode. An exemplary network 105 may include network attached storage (NAS), storage Area Network (SAN), Content Delivery Network (CDN), or other form The computer or machine readable medium may include a mass storage device.

[0044] In some embodiments, the exemplary server 106 or the exemplary server 107 may be a A web server (or set of servers) running a work operating system. Examples include, but are not limited to, Microsoft Windows Server 2008 R2 rver, Novell NetWare, or Linux. In an embodiment, the exemplary server 106 or the exemplary server 107 may be a cloud and / or network The present invention may be used for and / or provide a network computer. Although not shown, in some embodiments, the exemplary server 106 or the exemplary server The 107 supports email, SMS messaging, text messaging, advertising content, The exemplary server 106 may also include connections to external systems such as a content provider. Any feature may be implemented in the exemplary server 107, and vice versa.

[0045] In some embodiments, one or more of the exemplary servers 106 and 107 may include a limited Not used, but authentication servers, search servers, email servers, social networking servers Service server, SMS server, IM server, MMS server, exchange server, photo sharing server Service servers, advertisement servers, financial / banking related service servers, travel service servers, any suitable similar service for users of member computing devices 101-104. It may be specially programmed to run as a base server.

[0046] In some embodiments, and optionally, any of the embodiments described above or below. In combination, for example, one or more of the exemplary computer member devices 102-10 4. The exemplary server 106 and / or the exemplary server 107 may be implemented using a scripting language, a remote Route procedure calls, emails, tweets, and short message services (SMS ), Multimedia Message Service (MMS), Instant Messaging (IM ), Internet Relay Chat (IRC), mIRC, Jabber, Applications Simple Object Access Programming Interface (SOAP) ess Protocol) method, CORBA (Common Object Re quest Broker Architecture), HTTP(Hypertex t Transfer Protocol), REST (Representation al State Transfer), or any combination thereof. A specially programmed software module that can be configured to transmit, process, and receive The device may include a module.

[0047] FIG. 2 illustrates another exemplary computer-based system in accordance with one or more embodiments of the present disclosure. 2 shows a block diagram of a system / platform 200. Not all of the components are required to implement one or more embodiments. Changes in the arrangement and type of components may occur without departing from the spirit or scope of the various embodiments of the present disclosure. In some embodiments, the member computing device 202 shown Each of the sub-sub ... A computer-readable medium, such as a random access memory (RAM) 208 coupled to the In some embodiments, the processor 210 may include In some embodiments, the processor may execute computer-executable program instructions. 210 may include a microprocessor, an ASIC, and / or a state machine. In this embodiment, the processor 210, when executed by the processor 210, instructions that can cause processor 210 to perform one or more steps described herein. The present invention relates to a computer-readable medium that stores, processes, and / or communicates with a computer. In some embodiments, examples of computer readable media include, but are not limited to, 202a to provide computer readable instructions to a processor, such as processor 210 of client 202a. The information may include any electronic, optical, magnetic, or other storage or transmission device capable of storing or transmitting data. In some embodiments, other examples of suitable media include, but are not limited to, floppy disks. , CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, ASIC, structure any optical medium, any magnetic tape or other magnetic medium, or computer Any other medium from which a processor can read instructions. A computer-readable medium in the form of may transmit or carry instructions to a computer, The router, private or public network, or other transmission, both wired and wireless, In some embodiments, the instructions may be written in a language other than C, C++, V, isual Basic, Java, Python, Perl, JavaScript, etc. The code may include code from any computer programming language, including

[0048] In some embodiments, the member computing devices 202a through 202n may include a mouse, CD-ROM, DVD, physical or virtual keyboard, display, or other input In some embodiments, the system may include a number of external or internal devices, such as power or output devices. Examples of computing devices 202a through 202n (e.g., clients) are The network 206 may be any type of processor-based platform. This includes, but is not limited to, personal computers, digital assistants, personal digital assistants, Tablet assistant, smartphone, pager, digital tablet, laptop These include computers, Internet appliances, and other processor-based devices. In some embodiments, member computing devices 202a through 202n may be configured to communicate with one another as described in detail herein. in one or more application programs according to one or more of the principles / methodologies described In some embodiments, the member computing device 202a 202n is a registered trademark of Microsoft(TM), Windows(TM), and / or L Linux or other browsers or browser-enabled applications. In some embodiments, the illustrated media may be The member computer devices 202a to 202n are, for example, Microsoft's Intel rnet Explorer(TM), Apple Computer's Safari (TM), Mozilla Firefox, and / or Opera browser apps The device may include a personal computer running an application program. In this embodiment, the user 202 can access the member computer client devices 202a to 202n via the member computer client devices 202a to 202n. 12a through 212n communicate with each other and / or the network via an exemplary network 206. The network 206 may communicate with other systems and / or devices coupled to the network 206. As shown, exemplary server devices 204 and 213 are also coupled to network 206. In some embodiments, one or more of the member computing devices 202a-202n may be a mobile client.

[0049] In some embodiments, at least one of the exemplary databases 207 and 215 A database is a database managed by a database management system (DBMS). In some embodiments, the exemplary D The BMS management database is responsible for the organization, storage, and management of data in each database. The engine may be specially programmed to control the processing and / or search. In an embodiment, the exemplary DBMS manages databases for queries, backups and replication, rules, and implementing rules, providing security, calculations, performance changes and access logging, and / or It may be specially programmed to provide the ability to perform dynamic optimization. In this embodiment, the example DBMS managed databases are Oracle Database, IBM D B2, Adaptive Server Enterprise, FileMaker, Microsoft Access, Microsoft SQL Server, My You may choose from SQL, PostgreSQL, and NoSQL implementations. In this embodiment, the exemplary DBMS managed database is a hierarchical model, a network model, Relational model, object model, or field, record, file and / or any other method that may result in one or more applicable data structures that may include objects. In accordance with the particular database model of this disclosure, which may include any other suitable mechanism, an exemplary D Special care must be taken to define the respective schema for each database in the BMS managed database. In some embodiments, an exemplary DBMS managed database may be , may be specially programmed to contain metadata about the data that is stored.

[0050] In some embodiments, the exemplary novel computer-based system / platform of the present disclosure Exemplary novel computer-based devices, and / or exemplary novel computer-based systems. The computer-based components include, but are not limited to, IaaS (infrastructure ture a service), PaaS (platform as a servi) ce), and / or SaaS (software as a service) 3 and 4. The present invention may be specifically configured to operate on a cloud computer / architecture. 4 is an exemplary novel computer-based system / platform of the present disclosure, Exemplary novel computer-based devices and / or exemplary novel computer-based components A cloud computer / artifact may be configured to operate on the cloud computer / artifact. 1 shows a schematic of an exemplary implementation of the architecture.

[0051] In particular, exemplary computer-based system and / or device embodiments of the present invention To improve the accuracy and understanding of generalized linear models for benchmark studies, Outlier Bias Reduction (DOBR) may be used. However, this is only applicable if one or more It is a method that can be applied to a wide variety of analytical models in which there are independent variables and one dependent variable. The present disclosure and its embodiments provide a novel method for improving the accuracy of machine learning model predictions using DOBR. This illustrates a typical application.

[0052] In some embodiments, DOBR is not a predictive model. Instead, in some embodiments So this is a prediction model or interpretation model that can improve the accuracy of the model prediction. In some embodiments, the outliers identified in DOBR are Based on the difference between the supplied objective variable and the model-calculated value. When an outlier is identified, a pre-determined Through the selection criteria, outlier dependent data records and model-generated dependent variables are excluded from the analysis. These records may be permanently removed for further analysis. However, However, in other embodiments of the exemplary novel systems and methods, the deviations are calculated for each iteration of the model. The value identification process uses the predictive model of the last iteration defined by its calculation parameters. For each, include the entire dataset so that all records are subject to outlier scrutiny. Exemplary embodiments of the present invention may, for example, reduce the propagation of selection bias in the training data. To achieve this, we use the entire dataset in each iteration to minimize the impact of machine learning models. This reduces the bias in machine learning models, thus making them more accurate and efficient to train and and can be implemented to improve the operation of machine learning systems.

[0053] FIG. 5 illustrates an exemplary novel low bias method in machine learning, in accordance with one or more embodiments of the present disclosure. A block diagram of the reduction system is shown.

[0054] In some embodiments, the bias reduction system 300 may include, for example, a machine learning engine. Components for Dynamic Outlier Bias Reduction (DOBR) in the dataset under analysis In some embodiments, the DOBR may include an outlier according to a predefined criterion. Provides an iterative process for removing records. The criteria is expressed as a percentage. This is based on the insights and Indicates the amount of error the user is willing to tolerate in the model, potentially based on the results of other analyses. Value 100% means that all errors are tolerated and no records are removed during the DOBR process. Selecting 0% will remove all records. Generally, industrial use requires a value between 80 and 100%. A 95% margin of error has been observed.

[0055] In some embodiments, the user may use the display device 312 and the user input device 30 Among other user interaction actions using 8, the error tolerance can be achieved via the user input device 308. The bias reduction system is adapted to input values ​​and display results via the display device 312. Based on the error tolerance, the bias reduction system 300 may interact with the bias Data received in a database 310 or other storage device in communication with the reduction system 300 The bias reduction system 300 may analyze the set 311. and dynamically recalculating the received data set 311 to improve accuracy and efficiency. The predictions may be made using one or more machine learning models with outlier bias reduction.

[0056] In some embodiments, the bias reduction system 300 includes, among other components: , e.g., storage devices and memory devices, caches, buffers, buses, input / output (I / O) interfaces Interfaces, processors, controllers, networking and communication devices, operating Hardware and software computing, including operating systems, kernels, device drivers, etc. In some embodiments, the processor 307 may include a combination of other components. To implement its functionality, a component communicates with multiple other components. In some embodiments, each component may require a processor to perform the component function. 307, but in some embodiments, each component The events are scheduled to one or more processors in the processing system of processor 307. In other embodiments, each component has its own processor contained therein. .

[0057] In some embodiments, the components of the bias reduction system 300 include, among other components: Among other components, for example, the model index 302 and the model library 303 are The DOBR engine 301, the regressor parameter library 305, and the classifier parameter library 306 The components may include a data library 304, and a DOBR filter 306. Among other components, memory and storage devices, processing devices, communication devices, input Output (I / O) interfaces, controllers, networking and communication devices, operation Component functions such as rating systems, kernels, device drivers, and instruction sets It may include a combination of hardware and software for implementation.

[0058] In some embodiments, the DOBR engine 301 instantiates a machine learning model. The DOBR engine 301 includes a model engine for synthesizing and executing the model index. The model is then used to instantiate the model in a model library 303. For example, the model library 303 may access the DOBR engine 301. and the like. In some embodiments, the model library 303 may include a library of learning models. Among other possible classifiers and regressors, e.g., support vector machines (S VM), linear regressor, Lasso model, decision tree regressor, decision tree classifier, random Random Forest Regressor, Random Forest Classifier, K Nearest Neighbor Regressor, K Nearest Neighbor Classifier , gradient boosting regressors, gradient boosting classifiers, and other machine learning models. For example, the model library 303 may obtain a model according to the following exemplary pseudocode 1: can be imported. [Table 1]

[0059] However, in some embodiments, the machine learning models in the model library 303 To facilitate access to the library, the DOBR engine 301 Index each model with a model identifier that is used as a function by engine 301. For example, a model index 302 may be used to identify a regression model and a classification model. Among the various algorithms, for example, linear regression, XGBoost regression, support vector regression, Las so, K-nearest neighbor regression, bagging regression, gradient boosting regression, random forest regression, Models, including decision tree regressions, are indexed by a number identifier and labeled by name. For example, the following pseudocode 2 can be used with the model index 302: An example of a model index code that may be used is shown below. [Table 2]

[0060] Alternative implementations of pseudocode for model library 303 and model index 302 In some embodiments, the software instructions may be The library 303 or model index 302 is stored in the memory of the processor 307. In some embodiments, the DOBR error is buffered in a cache to provide to the The engine 301 accesses or calls the index via a communication and / or I / O device. The model index 302 may be utilized by, for example, To call a model as a function from the model library 303 via It may be used.

[0061] In some embodiments, the optimization of the model called by the DOBR engine 301 To facilitate ease of implementation and customization, the bias reduction system 300 can be implemented using a number of storage devices and and memory devices, such as hard drives, solid-state drives, random access The model parameters are stored in a memory or storage device such as a random access memory (RAM), a flash storage device, etc. For example, the regressor parameters may be recorded in the Regressor Parameter Library. The regressor parameters can be stored and adjusted in the library 305. 305 may store multiple parameters for multiple regressors, e.g., in real time. Storage and communications hardware configured with sufficient memory and bandwidth to For example, each regression instantiated by the DOBR engine 301 may include For the machine learning model, each parameter is defined in the regressor parameter library 305. In some embodiments, the user may input the user input device 30. 8, an initial set of parameters may be established. In some embodiments, the initial set of parameters may be predetermined or may be randomly generated. During instantiation of the regression machine learning model, the DOBR engine 301 ,The model identified by the model index 302 is,regressor parameter library 3 05. For example, the DOBR engine 30 1, for example, according to an identification (ID) number associated with a given regression model. For example, the regressor parameter library 305 may call the following set of parameters: The parameters of each regression model may be identified similar to pseudocode 3 below. [Table 3-1] [Table 3-2] [Table 3-3] [Table 3-4] [Table 3-5]

[0062] Similarly, in some embodiments, the classifier parameters are stored in a classifier parameter library The classifier parameters may be recorded and adjusted in the classifier parameter library 304. 4 stores and adjusts a large number of parameters for multiple regressors, e.g., in real time. Storage and communications hardware configured with sufficient memory and bandwidth to coordinate and communicate For example, each classification machine instantiated by the DOBR engine 301 may include For the learning model, the regressor parameter library 305 The meter may be initialized and updated. An initial set of parameters may be established via the user input device 308. However, In some embodiments, the initial set of parameters may be predetermined. During instantiation of the learning model, the DOBR engine 301 The model identified in 2 is used as a set of parameters in the regressor parameter library 305. For example, the DOBR engine 301 may correlate the data with a given regression model. A set of parameters may be called according to an identification (ID) number associated with the model. For example, the regressor parameter library 305 may contain a function similar to the following pseudocode 4: Parameters for each regression model may be identified. [Table 4-1] [Table 4-2]

[0063] In some embodiments, the model library 303 is accessed via the model index 302. It receives a set of models from the regressor parameter library 305 and and / or by receiving the respective parameters from a classifier parameter library 304. Thus, the DOBR engine 301 may, for example, use a cache or buffer of the DOBR engine 301. A user may load one or more instantiated and initialized models into the buffer. In some embodiments, the data set 311 is then extracted from the database 310, e.g., The same or a different cache or buffer or other storage device of the DOBR engine 301 Then, the processor 307 in the DOBR engine 301 or one of the processors The processor may execute each model to, for example, generate an activity based on certain input attributes associated with the activity. The data set 3 is used to predict each of the activity-related data values ​​that characterize the outcome or parameter of 11 may be converted, for example, to the energy usage of appliances in a home and / or commercial environment. , concrete compressive strength in various applications and compositions, object or image recognition, voice recognition, or are other machine learning applications. For example, the DOBR engine 301 uses, among other factors, Energy management of equipment based on historical energy usage, time, time, and location data311 The DOBR engine 301 may model the amount of fuel used. The model is retrieved from the model library 303 via the model index 302 connected to the bus. The DOBR engine 301 may then call the set of gradients. In the regressor parameter library 305 connected to the bus of the gin 301, Call the parameter file or log associated with the regressor for energy usage estimation. The DOBR engine 301 may then use the processor 307 to generate the model and based on model parameters, time and date, location, or other factors and combinations thereof. Future energy consumption may be predicted based on the above.

[0064] Similarly, for example, the DOBR engine 301 is designed to reduce the amount of concrete used, among other factors. Based on the data set of time, location, humidity, hardening time, and age, The DOBR engine 301 may model the compression strength of the DOBR engine 30 1 from the model library 303 via the model index 302 connected to the bus The set of regressors may then be called. The regressor parameter library 305 connected to the bus of the engine 301 For the RC compressive strength estimation, call the parameter file or log associated with the regressor. The DOBR engine 301 may then utilize the processor 307 to Concrete composition, time and date, location, or other factors and combinations thereof Based on the model and the model parameters, a future concrete compressive strength may be predicted. .

[0065] As another example, the DOBR engine 301 may take into account, among other factors, speech and ground Speech recognition may be performed based on a dataset 311 of truth speech transcriptions. The R engine 301 is connected to the bus of the DOBR engine 301 with model index 3. A set of classifiers may be called from the model library 303 via 302. The DOBR engine 301 receives the classifier parameters connected to the bus of the DOBR engine 301. In the data library 304, parameter files associated with classifiers for speech recognition are The DOBR engine 301 may then call up the processor 307. based on the model and model parameters for a set of one or more utterances using A transcription of the recorded audio data may be predicted.

[0066] As another example, the DOBR engine 301 may be any of the engines described in U.S. Pat. No. 10,339,695. As noted above, multiple renderings across imaging and / or visualization, among other factors, Rendering for medical images based on the dataset 311 of ring parameter settings The DOBR engine 301 may automatically predict the binning setting. A model is selected from a model library 303 via a model index 302 connected to the bus. The DOBR engine 301 may then call the set of classifiers. Rendering the settings in the classifier parameter library 304 connected to the bus 301 The parameter file or log associated with the classifier may be called to perform the The DOBR engine 301 then utilizes a processor 307 to generate one or more medical data. A rendering configuration data set based on a model and model parameters for a set of the data sets is The data may be predicted.

[0067] As another example, the DOBR engine 301 may be any of the engines described in U.S. Pat. No. 10,317,854. As stated above, among other factors, the results of the machine control commands and the Even if the robot control of the machine is performed based on the data set 311 of the simulation results, The DOBR engine 301 may include a model interface connected to the bus of the DOBR engine 301. A set of regression models is called from the model library 303 via index 302. Next, the DOBR engine 301 may be connected to the bus of the DOBR engine 301. Regression models for robot control in the Regressor Parameter Library 305 The DOBR engine 301 may then call the associated parameter file or log. Utilizing a processor 307, the set of control commands, environmental information, sensor data, and and / or based on models and model parameters relating to the simulation of commands, The success or failure of a certain control command may be predicted.

[0068] In some embodiments, the bias reduction system 300 may, for example, You may also implement machine learning models in a cloud environment as a cloud service. As described in US Patent No. 10,452,992, such cloud services It has a large number of users, a wide variety of algorithms and problem sizes, including those mentioned above, and Consider other potential models, datasets, and parameter tuning specific to your use case. In one embodiment, the bias reduction system 300 is implemented The services provided provide a number of programming interfaces (application programs, A programming interface (API, etc.) may be defined that allows users to without investing a lot of time and effort in tuning or learning advanced statistical or artificial intelligence techniques. It guides non-expert users to start using machine learning best practices relatively quickly. The interface allows non-experts to, for example, build and train machine learning models. default settings or parameters for various aspects of the procedures used to It may be possible to make the default dependent on the meter, where the default is similar to the individual user. For the model, the classifier parameter library 304 and / or the regressor parameters The default is derived from one or more sets of parameters in the data library 305. The settings or parameters are input by the user via the DOBR engine 301 and the optimizer 306. Customize your machine learning models using training on your datasets At the same time, the user may process the input records, It is expected to be used for many types of machine learning tasks, including feature processing, model building, execution, and evaluation. In at least some embodiments, the user may customize desired parameters or settings. uses predefined libraries that implement various types of machine learning tasks. In addition to or instead of the cloud service bias reduction system 300, for example, Extensible integration of services by registering customized features with the service The client implementing such customized modules or functionality may Modules may be shared with other users of the Service depending on Your business needs or goals. In some cases, the use of customized modules is restricted to their implementers / owners. There may also be cases where this is possible.

[0069] In some embodiments, the system may be implemented as a cloud service, a local system, or a remote system. whether it is implemented in a specific system architecture or in another system architecture. Regardless, the bias reduction system 300 is described in U.S. Pat. No. 9,646,262. This allows for an ensemble approach to training and deploying machine learning models. Such an approach may include models for the electronic It may be useful for application in data analysis using electronic data sets of activity data. In one embodiment, database 310 may include one or more structured or unstructured data sources. In certain embodiments, the unsupervised learning module may include, for example, a model library. We use multiple unsupervised learning techniques to model unstructured data in an ensemble of 303 models. The method is configured to assemble the training data set into a systematic data set. No learning module Organize unstructured datasets Multiple versions of a dataset while the supervised learning module is configured to assemble In this paper, we propose a method to generate a dataset of multiple versions, each of which is based on one or more machines. Generate a training ensemble and use the DOBR engine 301 and optimizer 306. After training each model in each ensemble, we then compare them according to the model error, for example. The method is configured to determine which machine learning ensemble of the trained models exhibits the highest predictive performance.

[0070] DOB for controlling hardware to make predictions based on a dataset 311 An example of an R Engine 301 instruction is shown in Pseudocode 5 below. [Table 5-1] [Table 5-2] [Table 5-3] [Table 5-4] [Table 5-5] [Table 5-6]

[0071] However, in some embodiments, outliers in the data set 311 are implemented This reduces the accuracy of the model and therefore the suitability of a given model in a given application. Increase the number of training iterations to achieve the correct set of parameters for To improve accuracy and efficiency, the DOBR engine 301 may A DOBR filter 301b is included to dynamically test data point errors to determine outliers. Therefore, deviations may be required to provide a more accurate or representative data set. In some embodiments, the DOBR filter 301b may be configured to remove predefined values. The selected criteria, e.g., those described above, may be entered by the user via the user input device 308. A regression analysis to remove outlier data points subject to a user-defined error tolerance provided by In some embodiments, the user-defined error tolerance is a percentage. For example, a value of 100% means that all of the error is tolerated, and the filter 301b means that no data points are removed, whereas, for example, a value of 0% means that no data points are removed. In some embodiments, the filter 301b is For example, the filter may be configured with an error tolerance ranging from about 80% to about 95%. 301b may be configured to perform the functions shown in pseudocode 6 below. [Table 6]

[0072] In some embodiments, the DOBR filter 301b cooperates with the optimizer 306. It operates by using a regressor parameter library 305 and a classifier parameter library For each model in library 304, the error is determined and the parameters are optimized. Thus, in some embodiments, the optimizer 306 is configured to The DOBR engine 301 may then determine an error rate and communicate the error to the filter 301 b of the DOBR engine 301 . Thus, in some embodiments, the optimizer 306 may, for example, 311 and model predictions are received and, among other error measures, such as outliers, convergence, and error A storage device and / or memory having sufficient memory capacity and bandwidth to determine the absolute error. For example, the optimizer 306 may include the following pseudo code: 7. The device may be configured to perform functions such as those shown in FIG. [Table 7]

[0073] In some embodiments, the bias reduction system 300 then, for example, displays 3 12, more accurate due to the reduction of outliers that would otherwise bias the predictions. and in an efficient manner, among other data generated by the DOBR engine 301. , machine learning model predictions, outlier analysis, and convergence of predictions may be returned to the user.

[0074] FIG. 6 illustrates a flowchart of an exemplary novel methodology in accordance with one or more embodiments of the present disclosure. show.

[0075] The DOBR engine 301 and the DOBR filter 301b described above are implemented using predefined Provides an iterative process for removing outlier records subject to a criterion. This condition is , a user-defined error tolerance expressed as a percentage. This is The user may allow the model to be modified based on the insights and other analytical results described in the A value of 100% indicates that all errors are tolerated and will not be recorded by the DOBR process. Selecting 0% means that all records will be removed. In industrial applications, error tolerances in the range of 80 to 95% have been observed.

[0076] However, in some embodiments, if the dataset does not contain outliers, the DO Note also that BR does not provide a value. However, in a real situation, It is rare for an analyst to have this knowledge before proceeding. As such, embodiments of the DOBR method use the percentage of the data set that represents model outliers. This pre-analysis step can also determine the appropriate error tolerances or outliers. This can be useful if a value exists.

[0077] The following steps outline the basic DOBR method applied to the full data set: .

[0078] Pre-analysis: In one embodiment, we first select an error tolerance criterion, e.g., ∝=80%. (How to determine this value from the data will be shown after the DOBR method is explained.) Next, define an error tolerance criterion C(∝) according to, for example, Equation 1 below.

number

[0079] where ∝ is the error tolerance criterion, C is a function of the error tolerance criterion, and f() is a comparison function. where y is the value of the data record, and y pred is the predicted value, and y tar is the target value.

[0080] Although other functional relationships may be used to set C(α), the percentile function is To understand why a given data record such as It is an intuitive guide.

number

[0081] Here, P ∝ is the percentile function, i is the index of the record entry, , m is the number of record entries.

[0082] Since the DOBR procedure is iterative, in one embodiment, a convergence criterion is also defined, where: It is set at 0.5%.

[0083] In one embodiment, the data set {x,y tar}404, solution model M408, and error tolerance criteria Given ∝424, DOBR reduces the bias in training the model M408. In some embodiments, the solution model M408 may be implemented to reduce, for example, In one embodiment, the model engine includes a processing device and memory and / or storage device. According to an embodiment, the exemplary methodology calculates the model coefficients M(c) 402 for all records. and the model estimate {y pred}410, for example according to Equation 3 below, for the complete input data tset {x,y tar}404 is applied with solution model M408.

number

[0084] Here, 0 indicates the initial state and x indicates the input record.

[0085] Then, according to an exemplary embodiment, the total error function 418 is calculated, for example, according to Equation 4 below: Calculate the initial model total error e0.

number

[0086] where e0 is the initial model total error and 0 is the initial value.

[0087] Next, according to an exemplary embodiment, the error function 412 is modeled, for example, according to Equation 5 below: Calculate the loop error.

number

[0088] where E is the predicted record error and k represents the iterations of record selection.

[0089] Then, according to an exemplary embodiment, the error function 412 is calculated as follows, for example, according to Equation 6: The new data record selection vector {I k}.

number

[0090] where I is the record selection vector.

[0091] Then, according to an exemplary embodiment, the data record selector 414 may use, for example, the following formula: 7, by selecting only those records whose record selection vector is equal to 1. Calculate the non-outlier data records that are included in the model calculation.

number

[0092] where in is an index that refers to the set of records included in DOBR as non-outliers. It is sex.

[0093] Then, according to an exemplary embodiment, the model 408 with the updated coefficients 402 is The new forecast value 420 and the model are calculated from the DOBR selection data record 416 according to the following formula 8: Calculate the delta coefficient 402.

number

[0094] Then, according to an exemplary embodiment, model 408 performs a complete run using the new model coefficients. Calculate new predictions 420 for the entire data set. This step is a formal step. This reproduces the calculation of the predicted value 420 of the DOBR selection record in the group, but in reality, for example, According to Equation 9 below, the new model can be applied to only the DOBR removed records.

number

[0095] Then, according to an exemplary embodiment, the sum error function 418 is calculated, for example, according to Equation 10 below: The total model error is calculated using the

number

[0096] where y^ is the target output.

[0097] Next, according to an exemplary embodiment, a convergence test 424 is performed, for example according to Equation 11 below: Test the convergence of the model.

number

[0098] where β is the convergence criterion 422, for example 0.5%.

[0099] In some embodiments, the convergence test 424 may be performed, for example, by checking whether the percentage error is, for example, If it is less than 0.5%, the iterative process may be terminated. Otherwise, the process may be restarted. Then, the steps above may be performed and the convergence criterion 42 may be satisfied. 2 may be retested. This process continues until the convergence test 424 fails the convergence criterion 424. Repeat until

[0100] FIG. 7 illustrates another exemplary computer-implemented method for reducing bias, in accordance with one or more embodiments of the present disclosure. 1 is a graph showing an example of a relationship between a model error of a database machine learning model and an error tolerance criterion. .

[0101] ∝ is an input parameter to DOBR, and model results vary based on the value selected. To obtain this, in one embodiment, a database procedure is used to justify which values ​​are used. It is important to document the procedure in which DOBR was developed and applied. There is no theoretical basis for that choice (yet). However, in practice, the model error vs. A plot of may result in a change in slope that reduces the apparent effect of outliers. For the calculation of the nonlinear regression 402 related to power generation benchmarking according to one embodiment of the present invention, The plot of is shown.

[0102] In one embodiment, the general shape of this curve is such that it always starts with the largest error at ∝=100%. , it is pre-determined that the model error is 0 when ∝=0%. In FIG. , note that the slope of the curve changes around ∝=85%. Also, for all smaller The slope is approximately constant for all ∝ values. The change in slope at this point corresponds to the removal of a data record. The variability of the model has not changed with respect to these error tolerance levels, or in other words This suggests that there are no outliers in the sample. There is an apparent slope change, which indicates that a certain proportion of the data set cannot be explained by the model. This visual test sets an appropriate error tolerance level. This can help determine whether a DOBR is needed. If the slope of the line in Figure 7 does not change, If the model fully explains the variability of the observed data, there are no outliers in the model. , there is no need to apply DOBR.

[0103] Simulation studies in which a certain percentage of additional variability was added to the data set In our study, curves like those in Figure 6 were generated approximately within the error tolerances programmed into the simulation. It shows an initial steep slope line that intersects with a smaller slope at a nearby point. In practice, when outliers are observed, the transition to a constant slope is generally gradual, and the model This suggests that there are multiple types of variation that are not explained by

[0104] Calculating appropriate error tolerances is a necessary part of using DOBR, which is important for model results. It also visually indicates the amount and severity of the influence of outliers on . This step documents the choice of ∝, When the impact of outliers is judged to be minimal compared to model predictions from outlier data could justify not using DOBR.

[0105] In some embodiments, the ∝ and model error vs. ∝ values ​​are determined to be the best for a particular scenario. Use as a metric to distinguish between models or ensembles of models for performance Different data sets may have different degrees of linearity, so it is important to test the model against the data. The exact value of ∝ can affect the performance of the model. Therefore, it is a function of the error tolerance level. We use the model error as a function of the variability of the data to form accurate predictions. Having some tolerance for model error allows you to determine how well a given model can account for the variability in the data. It is possible to determine the degree to which a model can explain a given set of variables. For example, the accuracy and precision of the model predictions can be determined by For example, models and / or model parameters that exhibit low model error for a high error tolerance. The model is adjusted by selecting the data and choosing a model that is more tolerant of outlier data. This may also be the case.

[0106] In some embodiments, model selection may involve, for example, variance of model errors and error tolerance criteria. A rule-based approach is used to identify the best performing model for a given dataset according to its It may be automated using programming and / or machine learning models. The model that best explains the outliers in a dataset can be automatically selected. For example, The errors may be compared between models for one or more error tolerances, with the model with the lowest error being chosen. The model with the difference is automatically selected to generate the prediction.

[0107] As a result, the DOBR machine learning technique according to aspects of the present disclosure provides more effective model training. Provides insight into data and model characteristics of individual datasets As a result, for example, artificial intelligence, data analytics, business intelligence and In other areas, machine learning models are becoming more effective and efficient with different types of data. Then you can decide which model is best for your application and type of data. To achieve this, the performance of models can be evaluated more efficiently. For example, in artificial intelligence applications The options were selected and trained using DOBR for the type of intelligence to be generated. Similarly, business intelligence and data analytics, as well as It provides a wide range of features, including: predicting logical behavior, recommending content, predicting resource usage, natural language processing, and more. Other applications, such as machine learning applications, use DOBR to refine model parameters. The model is adjusted based on the characteristics of the outliers and the model error corresponding to the outliers. It can be improved by selection.

[0108] FIG. 8 illustrates another exemplary computer-implemented method for reducing bias, in accordance with one or more embodiments of the present disclosure. 1 is a graph showing an example of a relationship between a model error of a database machine learning model and an error tolerance criterion. .

[0109] As an example of the implementation of DOBR on the California-Irvin dataset, The concrete compressive strength data set was downloaded from the machine learning data repository of the University of E. We use set 504. This data set contains 1030 observations with 8 independent variables. The first seven are concrete examples based on age. Composition, i.e., cement content, high performance water reducer, blast furnace slag, coarse aggregate, fly ash, fine aggregate, Describe aggregate, water, and age.

[0110] The output variable is the concrete compressive strength (Concrete C) measured in megapascals (MPa). For comparison, 1MPa is ≒ 145psi. The regression model is constructed, for example, according to Equation 12 below.

number

[0111] Here, a i are the coefficients calculated by the linear regression model, and x i are the observations of eight variables where i is the variable index.

[0112] Figure 8 shows the linear regression from 100 to 60% as a function of the DOBR error tolerance percentage ∝. It is constructed by running a regression model 504. ∝=100% to about ∝=95% Up to, there is a steep drop in the model error as a function of α, as shown by regression 506. The error reduction as a function of time decreases at a slightly lower rate until ∝=85%. From this point onwards, As shown in regression 508, ∝ decreases at a constant rate. The point at which the influence of model outliers is removed from the model calculation. In this case, the selection point is ∝=85%.

[0113] In one embodiment, the DOBR is then corrected and the linear regression model is The model is then rerun to determine the best fit to the non-outlier data. These calculations using the complete dataset 512 (Fig. 9) and the DOBR version (Fig. 10). The figure shows the results of a calculation in which outliers have been identified and removed from the calculation. The outlier value 516 is calculated from the non-outlier model. Both of these plots are shown in FIG. 10, the actual vs. predicted target values ​​are shown by diagonal lines 510 and 514, respectively, and the The full data set calculation (Figure 9) shows how the outliers are The DOBR correction plot (FIG. 10) shows that the diagonal line 514 is Bias was removed by bisecting the outlier518, and no obvious extraneous data were found that justify further study. A group of threshold data points 516 are shown.

[0114] FIG. 9 illustrates a non-bias-reduced reference computer in accordance with one or more embodiments of the present disclosure. 1 is a graph showing an example of the relationship between the predicted compressive strength of the base machine learning model and the compressive strength. do.

[0115] FIG. 10 illustrates another exemplary computer-implemented method for reducing bias, in accordance with one or more embodiments of the present disclosure. A graph showing an example of the relationship between the predicted compressive strength of the database-based machine learning model and compressive strength. It's rough.

[0116] Identifying outliers and the patterns they may form in plots of the above types. The outliers were useful for the additional benefit of the DOBR method in industrial applications. This information can form patterns or groupings that are simply not observed by conventional methods. It is created simply by using DOBR with the model provided by. Additional information or prerequisites In practice, the outlier set defined by DOBR is a measure of the improvement of the underlying model. These may provide information that is useful for improving, providing insight, or validating.

[0117] FIG. 11 illustrates a method for machine learning prediction with DOBR, in accordance with one or more embodiments of the present disclosure. FIG. 2 is a block diagram of another exemplary computer-based system.

[0118] In one embodiment of the present invention, the machine learning procedure uses n independent variables and records of length m. The first step is to start with a data set X^ consisting of an array (m×1) of objective variables Y^. To train a machine learning model, the dataset {X^,Y^} is computed as into two randomly selected subsets of a predetermined size according to One is to train the model and the other is to test its predictive accuracy. This is in order to do so.

number

[0119] where x^ is a subset of the independent variables X^ of the data set, and y^ is the independent It is a subset of the variable Y^.

[0120] In this study, a 70% / 30% split of {X^,Y^} was used for training (n records) and testing. j records (e.g., 70% of the records are training, 3 0% is test), e.g. 50% / 50%, 60% / 40%, 80% / 20%, 9 Any suitable training / test split such as 0% / 10%, 95% / 5%, or any other suitable training / test split A partition such as (x^,y^) can be used. train A machine learning model L is trained using For example, the predicted dependent variable {y pred} by computing the set This is tested.

number

[0121] In an exemplary embodiment, the model accuracy is then calculated by the norm ||y pred ,y test Measured as || , which may, for example, have the following form:

number

[0122] In an exemplary embodiment, in the training and testing environment, the input and output variables Since it has both, it is possible to measure outliers directly. Model predictions {y pred} outliers are those where the model L does not know the specific input This is because the input data for these records cannot be converted to a predictive value close to the objective variable. are factors and / or phenomena that the model cannot map onto the reality given the objective variables. The model coefficients are based on the assumption that all data records are equally valid. Because the calculations are done, keeping these records in the dataset could bias the results. .

[0123] In some embodiments, the DOBR process described above can be carried out, for example, with reference to FIG. , to find the best model that fits the data by removing outliers that would adversely bias the results. It operates on a given data set that the analyst wants to analyze. The initial data set with outliers removed Restricting the model solution to a subset of the data improves the predictive accuracy of the model. In an embodiment, the DOBR-assisted solution has two output results: a) a) the set of x-values, model parameters, and model solutions that describe the data; and b) the model The model is a set of x-values, model parameters, and model solutions that do not describe the data.

[0124] Therefore, in addition to computing more accurate models for limited data sets, Thus, in some embodiments, DOBR understands one or more causes of high model prediction error. To this end, we also provide an outlier data set that can be further studied for a given model.

[0125] In the exemplary embodiment of the machine learning framework presented earlier in this section, a predictive model The model is calculated from the training data and only that model is used in the testing phase. Since the accounting and testing phase does not need to use the target value to determine outliers, see Figure 6. In light of this, the DOBR method described above may not be applied. However, the DOBR method includes the There is an exemplary aspect that may not have been utilized in the previous example, namely the output of DOBR. The possibility of outlier-non-outlier classification suggested by the results is possible.

[0126] To illustrate DOBR in a machine learning application of one embodiment of the present invention, The dataset may be split into two randomly selected parts, one of which is the train In the training phase, the independent variables and the objective In the test, both the objective variable and the training variable are kept, but the objective variable is hidden and used to predict the objective variable. Independent variables are used. Known response variable values ​​are only used to measure the model's prediction error. .

[0127] In one embodiment, a training data set {x, y tar} train 604 Given a machine learning model L608 and an error tolerance criterion ∝622, DOBR can It may be implemented to reduce bias in the training of the machine learning model L608. In some embodiments, the machine learning model L608 may include, for example, a processing unit, a memory, and / or According to one embodiment, the exemplary method is implemented by a model engine that includes a storage device. The theoretical model is that for every record, train} is estimated, for example, according to the following Equation 16: We run the machine learning model L608 on the full input dataset {x,y tar} train Applies to 604.

number

[0128] Here, 0 indicates the initial state and x indicates the input record.

[0129] Then, according to an exemplary embodiment, the total error function 618 is calculated, for example, according to Equation 17 below: Then, calculate the initial model total error e0.

number

[0130] where e0 is the initial model total error.

[0131] Then, according to an exemplary embodiment, the error function 612 is modeled according to, for example, Equation 18 below: Calculate the delta error.

number

[0132] where E is the predicted record error and k represents the iteration.

[0133] Then, according to an exemplary embodiment, the error function 612 is calculated as follows, for example, according to Equation 19: Calculate the new data record selection vector.

number

[0134] where I is the record selection vector.

[0135] Then, according to an exemplary embodiment, the data record selector 614 may use, for example, the following formula: 20, by selecting only records whose record selection vector is equal to 1. , calculate the non-outlier data records to be included in the model calculation.

number

[0136] where in is an index that refers to the set of records included in DOBR as non-outliers. It is sex.

[0137] Next, according to an exemplary embodiment, machine learning module 60 with updated coefficients 602 8 is a complete trace using the DOBR selection data records, for example according to Equation 21 below. A new prediction 620 for the training set 604 is calculated.

number

[0138] Then, according to an exemplary embodiment, the total error function 618 is calculated, for example, according to Equation 22 below: , calculate the model total error.

number

[0139] Next, according to an exemplary embodiment, a convergence test 624 is performed, for example according to Equation 23 below: , to test the convergence of the model.

number

[0140] where β is the convergence criterion 622, for example 0.5%.

[0141] In some embodiments, the convergence test 624 may be performed, for example, to determine whether the percent error is, for example, If it is less than 0.5%, the iterative process may be terminated. Otherwise, the process We may return to the training data set 604.

[0142] In some embodiments, the DOBR iterative procedure is Instead of measuring how well a model can predict itself, we measure how well a model can predict how well a model can predict itself. The aim here is to test the model's ability to predict the outcome variable, and to select the By systematically removing records with low prediction accuracy, we focus on the majority of the data where the prediction is relatively good. This process must be done on the same dataset. If an outlier is identified in the test set, the record is removed from the training set. This process is important because it is the process where the new model (new model parameters) The DOBR method is similar in that after a calculation, records that were removed in a previous iteration are re-entered. This process requires the use of the same data set.

[0143] In one embodiment, this iterative procedure is performed after the learning model is defined. Based on the problem being solved, in one embodiment, the user selects a machine learning algorithm and then: Decide on the specific hyperparameters to "tune" or set for your model. These parameters The data may be selected using standard techniques such as cross-validation or based on specific user-specified can be selected by simply plotting the test error as a function of the parameter range of The particular value used is chosen to improve prediction accuracy while ensuring that the model is not under- or over-fitted. There are several robust tools available to assist in this process, User experience and intuition are also valuable advantages in selecting the best model hyperparameters. A particular model and associated hyperparameters are used in the examples described below.

[0144] The Error Tolerance vs. Model Error plot applies a sequence of error tolerances and tabulates the results. These plots are calculated from this step by plotting the , the contribution of these errors is slightly larger than the contribution of the errors in the data records that are fitted to the model. Identify the proportion of your data set that are outliers, in the sense that they are outliers. A plot of can show several types of variation that are not explained by the model. The slope is the gradient of the model. These variations are due to additional data not explained by the model. This can be useful for investigating the nature of the coded behavior. may be identified, and further investigation of these can help build more robust models. You can gain insight.

[0145] In one embodiment, two models are computed during training, as described above.

number

[0146] where {y ref} is the reference model used as a benchmark for measuring accuracy improvements. .

number

[0147] where {y Base} is constructed from the censored records of converged outliers and non-outliers Value data(x,y) in This is a DOBR-based model trained on

[0148] In some embodiments, the errors associated with Model 1 and Model 2 may be, for example, ε re f =||{y ref},{y test}|| and ε Base =||{y Base},{y test}||.

[0149] Thus, in some embodiments, the base model {y Base} means that it is a non-outlier record This suggests that it may be a better predictor. However, the test dataset It is uncensored and includes both non-outliers and outliers. Therefore, the customization of the non-outliers By applying the optimized model to the uncensored test data, we obtain ref} It is unclear whether a good predictive model will be generated. However, in many cases, Base is ε ref It can be observed that σ is either statistically equal to or greater than σ. do.

[0150] Non-machine learning apps where the goal is to calculate the best predictive model for a given dataset In this application, the DOBR model computed from selected (non-outlier) records is It always produces a lower model error because the identified outlier records are omitted. In the limiting case where there are no values, the data sets are the same, so the DOBR model error is the same for all models. This is equal to the delta error.

[0151] However, in machine learning applications, the goal is to model a subset of the available data. We develop (train) a model using the set of 100,000 samples, and then compare its predictive accuracy on a separate subset of the samples. However, in some embodiments, the test may involve measuring the DOBR. The method removes outliers from the model at each iteration before calculating the model parameters. In developing a model, this can be done in the training phase, but by definition the test target The baseline value can be used to measure the predictive accuracy of the model without sophisticated knowledge of outliers. This observation can be used to justify the standard DOBR method's DOBs calculated during the training phase. This means that more of the R model information can be used and generalized.

[0152] FIG. 11 illustrates another example of a method for reducing bias in machine learning, in accordance with one or more embodiments of the present disclosure. FIG. 1 is a block diagram of an exemplary computer-based system.

[0153] In some embodiments, as described above, during training, non-outlier (x,y) in D OBR select training dataset values, non-outliers {I in DOBR training for} Data selection vector, outliers(x,y) out DOBR selection training dataset for value, and outliers {1-I in}, the information of the DOBR training data selection vector for is generated.

[0154] In some embodiments, DOBR divides the training data into two mutually exclusive subsets. In addition, there is a corresponding selection vector, which can be expressed as, for example, Therefore, for each record in the training dataset, a classification (non-outlier or outlier) is performed. Provides a binary value.

number

[0155] In some embodiments, the training data attribute x train Complete set of DOBR raw materials Ingredient classification {I train} is, for example, a classifier machine learning model C stored in the model library 303. This model is used to build / train the training dataset Classify test data records as outliers or non-outliers based on DOBR probability knowledge For this, we use the test dataset x test For example, a classifier machine learning model C is It is implemented according to Equation 25 below.

number

[0156] Thus, in one embodiment, {I C} generates two test prediction datasets: Wow, x testin and x testout where I Ci = 1 or 0. Therefore, we have several possible "full dataset" prediction models for the analysis of the test dataset. In some embodiments, a model is created that shows the most predictive regressions for the entire data set. The three that showed positive results were as follows:

number

number

number

[0157] In some embodiments, for {y1}, the machine learning model L608 selects non-outlier data (x,y) in The DOBR test classification data is defined by Data x testinThe same procedure is followed for outlier data. In this situation, the goal of the combination is to find the most accurate predictive model with its corresponding dataset. In other words, this model uses the Overall predictive accuracy of the non-outlier and outlier models applied separately to each dataset Test the following:

[0158] In some embodiments, for {y2}, the machine learning model L608 uses the training data The DOBR test classification data x testin This model also applies to is L(x,y) train Using extensive knowledge of the DOBR-defined outliers and non-outliers x The goal of this model is to predict the target value of the non-external We test the predictive accuracy of the fully trained model applied separately to the benchmark and outlier datasets. The key is to strike.

[0159] In some embodiments, the third model {y3} is a combination of the predictive properties of the previous two approaches. This model is a hybrid that combines L(x,y) if any. train , Total tray The model trained on the learning,608,and L(x,y) out , each classified Outliers classified by DOBR on the training set applied to the dataset Test the predictive benefit of combining a particular trained model with the values. There are also additional hybrid models that could be explored in further studies.

[0160] In each of these three models and other embodiments, a complete test data set The prediction is done using both non-outlier and outlier records classified by DOBR. The ability of the DOBR method to improve the predictive accuracy of the entire machine learning model is demonstrated by the However, the main advantage of DOBR is that it identifies outliers in the model and The goal is to remove the outliers and calculate the best model predictor from the remaining non-outliers. According to the DOBR definition, outliers are the current variables, taking into account the machine learning model used. (or characteristics) that contain variations that are not adequately described.

[0161] In some embodiments, an outlier data set and a non-outlier data set are calculated. In this case, the analyst has three options. In one embodiment, the first option is to use the base model {y ref} and not applying DOBR. This is the risk tolerance vs. model error curve is close to a linear relationship. Apply one or more models, i.e. {y1}, {y2} or {y3}, combine the results and, for example, average In one embodiment, the third option is to average the predictions for only the non-outlier records. To develop measurements and modeling strategies for this specialized new data set, The goal is to further study the outlier data to improve the model, for example by modifying or explaining the machine learning model. For example, adding variables that are major factors in unexplained variation.

[0162] For option 3, there are multiple ways to calculate the non-outlier dataset, here Two possible choices are described. One reason for the relatively large number of possibilities is that many This may be due to the nonlinearity of the machine learning model used. C}*L[(x,y)train ,x tes t ]≠L[(x,y) train ,{I C}*x test ] This inequality is a problem that limits the complexity of many machine learning models. For example, in linear regression, an equation is applied, whereas in general machine learning models, Not as a rule.

[0163] In some embodiments, for non-outlier prediction, the DOBR method first performs a full dataset. The method was not designed to improve prediction of the Converge on the best set of outliers based on the model and dataset. Our calculations improve accuracy, but provide guidance on how to predict outliers The implied decision is to consider outliers that reflect unique data variation that is not present in the non-outlier model. The idea is to apply different models to the dataset.

[0164] In some embodiments, two models are defined to test non-outlier prediction accuracy: Remove outliers from the analysis. The first choice for selecting a non-outlier data set is For example, according to Model 6 below, the DOBR classification vector {I C} Reference model {y ref}.

number

[0165] In some embodiments, the reference model is test To make predictions from Then, we use the training data defined model. Based on the knowledge of the DOBR method, the classification vector is This model applies DOBR to the most general or broad area model.

[0166] In some embodiments, the second model is trained from non-outlier training data. By using the DOBR model created in stages, the narrowest or "accurate" In this way, we can model DOBR as follows, for example, in model 7 below: C} selected by applies only to records that have been

number

[0167] There are other models that can be formed from the analytical formulation developed in this study, depending on the problem. However, they could greatly improve predictability. The models used, {y4} and {y5}, are ranked according to training domain usage and model definition. This limits the cases to represent the broadest and narrowest versions.

[0168] In some embodiments, the DOBR development models defined above, such as models 3-7, To test the predictive accuracy of the models {y1}, {y2} and {y3} (models 3, 4 and 5, respectively), ) as a comparison standard for {y ref}. {y4} and {y5} (models 6 and 7, respectively) is the model prediction for the non-outlier dataset, and the comparison criterion is {I C}*y test Thus, in some embodiments, the error can be calculated, for example, as shown in Equations 26, 27, and 28 below: 8, where m is the length of the test data set and G is the number of non-outliers. is the length of the dataset).

number

number

number

[0169] In the following example of an exemplary embodiment, a measure of the predictive accuracy of DOBR is how much (if any) If ε1, ε2 and / or ε3 are ε Ref The non-outlier data set is measured by whether the For the set errors ε4 and ε5, the improvement measure is the outlier-adjusted base error ε Ref Error for The adjustments for the example results are explained below.

[0170] In some embodiments of the exemplary novel machine learning example of DOBR refinement, The accuracy of the five models is compared to seven machine learning regression models, namely, linear regression, k-nearest neighbors, L Tested using ASSO, support vectors, decision trees, bagging, and random forests. These machine learning regression models are examples of a wide range of model structures. In particular, Neural networks, clustering, ensemble models, and their combinations Additional or alternative models, such as the above, are also contemplated.

[0171] Linear regression is a method that gives analysts insight into a process, where coefficients (or are model parameters) can have technology / process related meanings. The model must be provided by the analyst, and the coefficients are the target values ​​for the prediction and the data. The supply target value is determined by minimizing the error between the supply target value and the actual supply target value.

[0172] “least absolute shrinkage and selection LASSO, short for “lasso operator”, is a regression-related method, where An additive term is added to the objective function. This term is the sum of the absolute values ​​of the regression coefficients and is determined by the user-specified The purpose of this summation term is to minimize the value of the variable (or feature) coefficients. The minimization keeps only the dominant coefficients, and It can help reduce the difficult-to-interpret effects of variable (or feature) covariance or collinearity.

[0173] Decision tree regression can mimic human thinking and is intuitive and easy to interpret. The rule selects a decision tree structure that logically shows how the x values ​​generate the target variable. Certain parameters such as maximum depth and minimum samples per frame are It is set by analysts in machine learning practice.

[0174] Random forest regression is based on decision tree methods. Just as a forest is made up of trees, In turn, a random forest regression model is composed of a group of decision trees. quantity (number of trees in the forest), similar to the maximum depth of a decision tree among multiple trees Some parameters, leaf characteristics, and model errors are calculated and applied. The forest structure is defined by providing relevant technical parameters.

[0175] k-NN stands for k-nearest neighbors, where the predicted value is the k-nearest neighbor in the x (or feature) domain. Select the metric that measures the distance and the specific number of nearest neighbors to use. The task is performed by the analyst when tuning the model for prediction on a given data set. This is a simple method suitable for predicting regression and classification. It is.

[0176] Support vector regression is a general-purpose machine learning method with multiple variations. Optimization means fitting a model to data, and optimization is usually the error between the predictor variables and the objective variable. In support vector regression, the error criterion is the error value ε ε is smaller than ε, we say that the approximation is sufficient and only errors larger than ε are measured. In addition to this attribute, the method is generalized to be easily implemented and optimized. Transform data into the nonlinear domain, possibly using a user-defined transformation function or kernel Multidimensional data structures allow us to model data in the traditional spirit of regression. It is used when the goal is to compute robust predictions rather than

[0177] In bagged regression, predictive estimates are calculated from drawings of random subsets with replacement. Each random sample computes a decision tree (default) prediction for the objective variable. The final ensemble Bull forecasts can be calculated in a number of ways, the average being one example. The machine learning variables are the number of estimators in each ensemble, the number of draws to train each estimator, The variables (or features) to be analyzed and the number of samples, as well as selection / replacement guidelines. can reduce variance compared to other methods such as decision tree regression.

[0178] Classifier model C[(Itraining ,x train ),x test ] is the number of non-outliers and and outliers are defined by the DOBR non-outlier / outlier classification and the training set x values. This is an example of how the training set is transformed into a test set. DOBR's machine learning application to transfer outlier knowledge to the production or production dataset This is a critical step in our application. Misclassification can reduce the accuracy of machine learning predictions. The utility of the DOBR method for improving

[0179] Decision trees, k-NN, random forests, and bagging classifier models are used to classify We tested the accuracy of the bagging and random forest models. The rule was adjusted to produce the correct error tolerance for non-outliers. Although classification accuracy is paramount, extensive classifier analysis may suggest other models. This is outside the scope of the earlier discussion.

[0180] FIG. 12 illustrates a biased method for predicting concrete strength in accordance with one or more embodiments of the present disclosure. The model error and error of some exemplary computer-based machine learning models with reduced 11 is a graph showing an example of a relationship with an acceptance criterion.

[0181] In the first example, the same data set as previously described for concrete compressive strength is used. where DOBR is applied to the full dataset. The dataset is analyzed as a function of its composition and exposure defined by eight quantitative input variables. The data set contains 1,030 records or instances. It has a database of machine learning algorithms and can be found in the Machine Learning Repository Archive at the University of California, Irvine. It can be attached.

[0182] In machine learning training practice, this dataset is split into a 70%:30% split, Here, model tuning is performed on the training dataset (70%) and predictions are The results were measured on the test dataset (30%).

[0183] The model adjustment results of the seven machine learning models for predicting concrete compressive strength are shown in the table below. Shown in 1. [Table 8]

[0184] The default model parameters (e.g. for Python 3.6) add information to the results. In some embodiments, the tuning process requires minimal training. Select the parameters to be optimized and test the data set error using the mean square error as a metric. More sophisticated algorithms could be applied, but the simple approach Use a procedure to ensure that the results fit both data sets fairly well. did.

[0185] In one embodiment, in order to apply DOBR, data with excessively large errors, if any, are In some embodiments, the machine learning model determines the percentage of the error. The model is applied to a sequence of probability distributions and the corresponding model error is recorded. This is only used to measure the predictive accuracy of the machine learning model, so it is not part of the training data. The percentage of data included in the model, the error tolerance " refers to the total amount of model error that the user is willing to tolerate, and is the The percentage of data that describes the

[0186] In some embodiments, the error tolerance percentage sequence ranges from 100% to 6 in increments of 2. The range is 0%.

[0187] FIG. 13 illustrates a bias for predicting energy usage in accordance with one or more embodiments of the present disclosure. Reduced model error and error tolerance for some exemplary computer-based machine learning models 11 is a graph showing an example of a relationship with a reference.

[0188] A second example includes home environment and lighting conditions along with appliance energy usage data, The pulling was done every 10 minutes for four and a half months. This resulted in the measurement of 29 attributes, namely: It consists of 28 input variables, 1 output (target variable), and 19,735 records. The dataset and documentation are available at the University of California, Irvine Machine Learning Repository. Can be found in ReArchive.

[0189] Similar to the above, in some embodiments, seven machine learning techniques for predicting energy usage of equipment are The model adjustment results are shown in Table 2 below. [Table 9]

[0190] In some embodiments, the default model parameters (e.g., for Python 3.6) The data are not shown as they do not add any information to the results. The adjustment process uses the average Use squared error to select parameters that minimize the error between the training and test datasets. More sophisticated algorithms could have been applied, but the simple approach Use a procedure to ensure that the results fit both data sets fairly well. did.

[0191] In one embodiment, in order to apply DOBR, data with excessively large errors, if any, are In some embodiments, the machine learning model determines the percentage of the error. The model is applied to a sequence of probability distributions and the corresponding model error is recorded. This is only used to measure the predictive accuracy of the machine learning model, so it is not part of the training data. The percentage of data included in the model, the error tolerance " refers to the total amount of model error that the user is willing to tolerate, and is the The percentage of data that describes the

[0192] In some embodiments, the error tolerance percentage sequence ranges from 100% to 6 in increments of 2. The range is 0%.

[0193] 12 and 13 partially illustrate the ability of the machine learning model to adapt to high variability data. The more linear the line, the better the model is at describing the data. This results in fewer, if any, outliers. The linear nature of the models means that they can describe the entire training dataset almost perfectly. The nonlinearity of the results of the energy data set indicates that the model makes inaccurate predictions or This indicates a high percentage of data records that produce outliers.

[0194] For example, linear regression530, LASSO540, decision tree regression522, and random forest Regression528, k-nearest neighbor regression524, Support Vector Regression (SVR)520 and Bagging The specific data plots shown above include the regression 526, as well as, for example, the linear regression 730. , LASSO 740, Decision Tree Regression 722, Random Forest Regression 728, k-Nearest Neighbor Regression The above, including regression 724, support vector regression (SVR) 720 and bagging regression 726. For each curve in the energy usage data plot, the deviations are calculated according to an embodiment of the present invention. The percentage of values ​​determines the error tolerance at which the error tolerance begins. A straight line defined by the radii can be extrapolated. This process can be automated, but in practice, the selected The error tolerances selected may be performed manually to ensure that they reflect the analyst's judgment.

[0195] The practice of extrapolation and the selection of an error tolerance are relatively simple processes, but have very important implications. This indicates how well the proposed model fits the data. The complement of is the percentage of the data set that are outliers, i.e., the percentage of This is the percentage of records for which a reasonably accurate prediction cannot be made given the data. Important considerations when selecting a machine learning model (or any model) for a given dataset and practical application Table 3 shows the error margins selected for each mode of the two example data sets. This indicates the capacity value. [Table 10]

[0196] In some embodiments, the predictive accuracy of only the DOBR selection values ​​is compared to a reference model. However, the method itself does not provide any specific information about the improvement of prediction accuracy for the complete dataset. This is the fundamental usefulness of DOBR because it is not possible to This improves the predictive ability of some of the datasets, but does not provide information about outlier records. This section deals with the issues that may arise as a trade-off for the analyst. The result selected by DOBR, if any, is compared to the test data prediction of the corresponding reference model. The question is how accurate it is compared to

[0197] The baseline error is calculated for the complete data set. The adjusted standard error value for is calculated by multiplying the full standard error by the error tolerance. For example, if the standard error is 10.0 and the error tolerance is 80%, the adjusted The standard error is 10 x 80%, or 8.0. In this interpretation, the definition of "error tolerance" is For example, if the non-outlier data is calculated for 80% of the data, the total error is 80% of the data should still remain non-outliers. This is the definition of the error tolerance.

[0198] The results of measuring the prediction accuracy performance of DOBR selected non-outliers are shown in Tables 4 and 5 below. For example, each of the concrete strength data set and the energy data set The standard error is the actual error tolerance percentage multiplied by ||{y Ref},{y test Point estimate of}|| The random forest classifier improves the prediction accuracy of DOBR. It was determined that this would not change the conclusions regarding the improvement in hardness, and therefore was not applicable to the concrete dataset. It was not used.

[0199] For all statistics below, the results are given for one subset of the training and test data. The mean ± 95% confidence interval from 00 random trial selections is shown. Some examples from the table below In order to manage the computation time issue, the support vector results are generated in fewer iterations. The results were calculated from the number of times (5 or 10). [Table 11] [Table 12]

[0200] As shown in Table 4, there is little improvement in prediction using DOBR selection records. The results are not surprising and, in fact, the error tolerance vs. model error curve shown in Figure 12 Predicted based on shape.

[0201] Also, as expected, based on FIG. 13, Table 5 shows the results for the bagging classifier and the random field. For both forest classifiers, there was a significant improvement in DOBR selection prediction from the reference model values. 14A and 14B, respectively, below. BR model {y5}={y 1in} shows the largest improvement, suggesting removal of outliers before model training, Combine with DOBR classification to simply use DOBR classification for full (non-DOBR) models. The difference in improvement between models indicates that model choice is important. This decision is made by the analyst, but it is important to compare the prediction accuracy of each model. It is interesting to note that the model run time and several other factors are also important. The study was not designed or intended to suggest that one model is more effective than another. Not as illustrated.

[0202] The conclusions in Table 5 are clear and statistically significant. As shown in a plot like Figure 13, Considering the possibility of outlier bias, the machine learning model using the DOBR method is It provides better prediction accuracy for non-outlier records than a machine learning model without Therefore, an exemplary new machine learning model with DOBR can be provided. New computer systems improve accuracy and reduce error in making predictions, This improves the performance and efficiency of implementing the model. However, the improvement comes at a cost. That is, no prediction or consideration may be given to identified outliers. In some embodiments, how the outlier records are modeled depends on the application. may vary based on

[0203] Table 6 shows the results of training the concrete compressive strength dataset using the bagging classifier. The prediction accuracy results for the test sampling are shown. The random forest classifier The table shows the training and test data sets. For 100 random selections of the set, there is a confidence level of 95% between the test data and each model. The root mean square error in % (see Equation 15) is shown. [Table 13]

[0204] In linear regression and LASSO, the error of the base model or reference model is the largest. However, However, the {y1} model shows statistically the same predictive accuracy as all other models except for decision trees. In the case of , the decision tree model produced the highest prediction accuracy, surpassing all other models except for linear regression and LASSO. Dell finds that adding DOBR will not improve things.

[0205] Table 7 shows, for example, the DOBR model: bagging classifier concrete compressive strength prediction accuracy. Increase (+) or decrease in the prediction accuracy of the DOBR model relative to the reference model in each case of Indicates (-). [Table 14]

[0206] The model error vs. error tolerance curves for linear regression and LASSO are plotted with maximum nonlinearity. The others are approximately linear, meaning the model predicts the objective variable well and no outlier analysis is required. These results are not surprising, as they suggest that The model output for the predicted concrete compressive strength is given in the attachment. This is shown in Appendix A of the

[0207] Here, looking at the results of the energy consumption prediction error in Table 8, for example, bagging and random There are different situations involving equipment energy consumption prediction errors for forest classifiers. The linear regression and LASSO models had the largest reference prediction error, while the random forest The DOBR model error in the right three columns is often smaller than that of the DOBR model. The results show that the model produces higher prediction accuracy than the reference model. [Table 15]

[0208] Interestingly, the bagged reference model has the largest baseline error value, but its DOBR extension The results of the Zhang model are generally in the same statistical range as other models. Also, for practical reasons, The support vector model was run for only 10 iterations, which is consistent with the overall model results. Account for the increased uncertainty in

[0209] For example, we present detailed improvements in the performance of the DOBR model's energy consumption prediction accuracy. 9. At least one DOBR model has been shown to be superior to most machine learning models. Note that this provides some improvement in prediction accuracy. However, there are still significant differences, so conclusive results cannot be drawn regarding the improvements in predictability brought about by DOBR. The model error vs. error tolerance curve for the energy data shows that all plots The random forest and decision tree models have the smallest curvature. And models, especially random forests, are likely to be more effective based on the results presented here. It turns out that this variation can be modeled well. The model output is shown in the attached Appendix B. [Table 16]

[0210] 14A and 14B are diagrams illustrating a DOBR classifier according to one or more embodiments of the present disclosure. According to an exemplary embodiment of the exemplary computer-based system of the present invention, Figure 1 shows plots of the non-outlier and outlier distributions in the sample.

[0211] The concrete data set is relatively small, so visual insights can be gleaned from data plots. However, in this case DOBR has little value, so we will not use this data set. Graphing the energy consumption does not improve our understanding of the operation of the DOBR. For the prediction of the Ghee dataset, DOBR significantly improved the prediction. However, Due to its relatively large size (13,814 training records, 5,921 (test records), it is difficult to interpret the direct scatter plot visualization. A scatter plot with such a large number of points can obliterate any detail. The best results are for the non-outlier dataset, but the DOBR method and the classification model The question remains as to how these results are produced.

[0212] In some embodiments, to address this issue, two model representations are used: {y4}, the random forest classifier (Figure 14A) and {y5} for the outlier and non-outlier datasets In one embodiment, the error distribution for the bagging classifier (FIG. 14B) can be analyzed. The error of the outliers should be smaller than the error of the outliers by design, but the novel DOBR model The model and classification process are built from training data only, so the test data set The data may contain previously unseen information, so that the model and classification calculations are correct. The results may not be accurate and the extent of the classification error can be visualized in these plots. As shown in Table 5, the two approaches have the largest and smallest improvement effects, respectively. Therefore, it is performed for linear regression models and bagged regression models.

[0213] For purposes of discussion, the baseline error values ​​are highlighted in both plots of FIG. 14A and FIG. 14B. The top set of arrows indicates that 80% of the non-outlier error values ​​are below 1,000. This indicates that 20% of the error values ​​are >1,000. The set of arrows indicates the outlier distribution, where approximately 20% of the outliers have an error of <1,000. or 80% had an error of >1,000, which represents the error of outliers. Without prior knowledge of the error tolerance percentage, it is difficult to accurately estimate the accuracy of the classification process. Although it is not possible to calculate the mean, the plot above shows that most values ​​are appropriate even if misclassification occurs. This indicates that the product is classified as

[0214] FIG. 15 is a diagram illustrating a method for predicting non-productive time in drilling a well, according to one or more embodiments of the present disclosure. An exemplary computer of the present invention having a DOBR trained machine learning model for For an exemplary use case of an exemplary embodiment of a computer-based system, the error tolerance 1 shows a plot of model error as a function.

[0215] Offshore well drilling operations present unique challenges for the oil and gas industry. Weather and depth In addition to observable logistical and environmental risks from the sea, there are hidden risks that operate in high temperature, pressure and vibration environments. Drilling time is kept on a tight schedule and downhole construction is Delays due to equipment failures (non-productive time or NPT) can result in large revenue penalties.

[0216] To help manage the NPT, machine learning models are being used to refine contract terms that set drilling targets. The goal is to include these estimated delays to help predict future downtime events. When you check the history events, you can see the drill distance [feet], hole size [in ch], tool size [inch], severity of position pressure, maximum dog leg [deg / 1 00 ft], vibration severity category, curvature category, and NPT(hr).

[0217] Linear, xgboost, gradient boosting, and random forest regression models A 0 / 20 training / test split was used to apply the model to downhole equipment failure data. Using Hyperband, we measured the model prediction accuracy shown in Table 10 below. and related parameter values ​​were adjusted. [Table 17]

[0218] The classification function that transfers the DOBR outlier information to the test dataset is, for example, equal to 5. The random forest model with the largest number of estimators can be selected as the model. This is also accomplished in the training part of the analysis. The metric for parameter selection is the training Calculate the proportion of correctly classified elements of the set and compare it to the error tolerance of the model. And so.

[0219] The analysis included linear regression, where the coefficients were the effect of additional best practice improvements. Because it is the only model that can provide the engineering insights that help identify the best Other models are more robust from a predictive standpoint but provide little insight. stomach.

[0220] As described in this specification, a basic DOBR process can be constructed. There are several relevant models. In this example, three are shown. M is the number of Represents a hypertuned model.

[0221] Using DOBR, we find the inliers of the training and test datasets. and outliers were selected. [Table 18]

[0222] where Data_xin_scrub and Data_yin_scrub are the DOBR calculations from the training set. are the inner values, Data_xout_scrub and Data_yout_scrub are the DOBs from the training set. R is the calculated outlier, and DOBR_Model_testin and DOBR_Model_testout are the DOBR calculation outliers, respectively. are the inner and outlier values ​​of the test dataset calculated from the class model, and BaseModel_yin ,BaseModel_yout is a non-DOBR classification model that classifies the inside values ​​and outliers. BR calculation model result, I is the BaseModel value for the DOBR definition inner value BaseModel_yi n and to BaseModel_yin for the DOBR defined outliers.

[0223] From these subsets, three DOBR models are obtained: a.DOBR_Model#1=[DOBR_Model testin ,DOBR_Model testout ] b.DOBR_Model#2=[BaseModel yin ,DOBR_Model testout ] c.DOBR_Model#3=[DOBR_Model testin ,BaseModel yout ]

[0224] Error tolerance percentage vs. model error curve for the hyper-tuned model above will produce the curves shown in Figure 15. An important property of these curves is their curvature , not the error value itself. In general, the slope of a given curve over the domain (0, 100%) is the line The more the shape of the data, the smaller the impact of outliers. The curve is linear up to about 80% of the error tolerance, then has a nonlinear slope of various degrees. When analyzing the slope as a function of the error tolerance, the following table (Table 11) shows the DOBR distribution. The error tolerance thresholds determined for the analysis are shown in Fig. 1. [Table 19]

[0225] All models use the computed hyperparameters and assigned error tolerances. The model output for predicted NPT is shown in the attached Appendix C. The difference results are shown in Table 12 below. [Table 20]

[0226] Now that we have the three DOBR models as well as the non-DOBR model, we can predict the future. In general, linear models are At this point, the model #1 or #2 has the lowest prediction accuracy. Analysts should consider these accuracy figures and other practical considerations, such as the model to be applied to future projections. This allows for a balance between the computation time required to select a model and the amount of computation time required to select a model.

[0227] The results of training and implementing a machine learning model using DOBR are shown in Fig. 1. Applications have been in the area of ​​shrinkage prediction and energy prediction, but other applications are conceivable.

[0228] For example, image rendering and visualization may be performed using the techniques described in U.S. Pat. No. 10,339,695. As described above, automatic prediction of rendering parameters based on medical data, for example, A machine learning model may be used to perform the content-based classification. It is also used to train and run machine learning models for rendering. A medical data set representing a three-dimensional volume of a patient may be used as input data. DOBR may be used to remove outliers from the training medical dataset, As a result, according to the DOBR technique described above, the machine learning model is trained on non-outlier data. Machine learning models can extract features from medical datasets and It outputs values ​​for two or more physically based rendering parameters based on the input of To achieve this, we performed deep learning on non-outlier data from a training medical dataset. In some embodiments, the physically based rendering parameters are trained using The meter is designed to provide consistent data processing, lighting design, view design, material validity, or internal rendering. The Physically Based Renderer uses the output values ​​obtained from the application of Use to render a photorealistic image of a three-dimensional area of ​​a patient.

[0229] Another exemplary application of DOBR for training and implementing machine learning models is As described in U.S. Pat. No. 10,317,854, a machine issues control commands. To generate control commands to control the vehicle, we use the DOBR technique described above to model the vehicle using machine learning. In such an example, the simulator may train the The simulator may perform a simulation of the operation of the machine based on the control command. By simulating the physical actions of a machine based on the input, the machine learning model A complete dataset for training the model can be generated. The operation data, control command data, etc. are used as inputs for each simulation. When training model parameters including data and machine data, outlier simulation is performed. may be processed using DOBR iterations to ensure that all vertices are removed. .

[0230] In some embodiments, and optionally, any of the embodiments described above or below. In combination, the exemplary DOBR machine learning model performs the following steps for the desired non-outlier data: Monte Carlo method of calculation algorithms that can take into account historical quality data (e.g., Solov ay-Strassen type algorithm, Baillie-PSW type algorithm, Mi Leller-Rabin type algorithm and / or Schreier-Sims type algorithm In some embodiments, and optionally, In any combination of the above or below described embodiments, the exemplary DOBR machine The machine learning model may, for example, but not limited to, include at least one machine learning technique (e.g., Not specified, but decision trees, boosting, support vector machines, neural networks algorithms, nearest neighbor algorithms, naive Bayes, bagging, random forest, etc.) Collected and / or compiled sensor data (e.g., environmental and / or cargo physical / By applying the In some embodiments, and optionally, any of the methods described above or below may be used. In any combination of embodiments, the exemplary neural network technique is not limited to the following: However, feedforward neural networks, radial basis function networks, and linear networks are Current neural networks, convolutional networks (e.g., U-net), or In some embodiments, the network may be one of the following: Alternatively, in any combination of the above or below described embodiments, An exemplary implementation of the network may be performed as follows. i) Define the neural network architecture / model. ii) forwarding the input data to an exemplary neural network model; iii) Incrementally training the exemplar model. iv) Determine the accuracy at a particular number of time steps. v) The exemplary trained model is used to process newly received input data. applies. vi) Selectively and in parallel, run the exemplary trained models with a given periodicity. Keep training.

[0231] In some embodiments, and optionally, any of the embodiments described above or below. In combination, the exemplary trained neural network model A neural network is modeled using at least one neural network topology, a set of activation functions, and connection weights. For example, the topology of a neural network can be specified as The message may include a configuration of nodes of a neural network and connections between such nodes. In some embodiments, and optionally, any of the embodiments described above or below. In combination, the exemplary trained neural network model Not specified, but may be specified to include other parameters, including bias values / functions and / or aggregation functions. For example, the activation function of a node may be a step function, a sine function, a continuous or discretized function. A linear function, a sigmoid function, a hyperbolic tangent function, or any other function that represents the threshold at which a node is activated. In some embodiments, and optionally, the above or In any combination of the embodiments described below, an exemplary aggregation function is It may also be a mathematical function that combines (e.g., sums, multiplies, etc.) multiple input signals. In an embodiment, and optionally in combination with any of the embodiments described above or below. In this case, the output of the exemplary aggregation function may be used as an input to the exemplary activation function. In some embodiments, and optionally, any of the embodiments described above or below. In combination, the bias is the aggregation function so that the node can be more or less activated. and / or may be a constant value or a function that may be used by the activation function.

[0232] In some embodiments, and optionally, any of the embodiments described above or below. In combination, exemplary connection data for each connection in the exemplary neural network is The data may include at least one of node pairs or connection weights. If the global network includes a connection from node N1 to node N2, an example of that connection is Explicit connectivity data is<N1、N2> In some embodiments, and optionally in any combination of the embodiments described above or below, The weights determine whether and / or how the outputs of N1 are modified before being input into N2. In the example of a recurrent network, A node may have connections to itself (e.g., connection data is <N1 , N1>).

[0233] In some embodiments, and optionally, any of the embodiments described above or below. In combination, the exemplary trained neural network model is It may also include identifier (ID) and compatibility data. For example, a variety of IDs may be used to identify multiple species (e.g. cargo damage). For example, the fitness data may indicate which species the model falls into (e.g., loss category). ,An exemplary trained neural network model is trained on the input sensory dataset For example, the goodness-of-fit data may indicate how well a model fits a given set of variables. The fitness function may include a fitness value determined based on an evaluation of the fitness function. For example, an exemplary fitness function is , an example trained neural network model for the input sensory dataset An objective function based on the frequency and / or magnitude of errors produced by testing the model. As a simple example, if the input sense data set contains 10 rows, Let the dataset contain two columns, denoted A and B, and let us consider an example trained neural Suppose the network model takes input values ​​of A and outputs the predicted values ​​of B. In this example Testing the exemplary trained neural network model in is to input each of the 10 values ​​of A from the input sensor data set, and predict B comparing the values ​​of B with the corresponding actual values ​​of B from the input sensor data set, and determining whether and / or how much the predicted and actual values ​​differ. As an example, suppose a particular neural network reduces the value of B to 9 out of 10 rows. For accurate predictions, the example fitness function has a 9 / 10=0.9 fit to the corresponding model. The above examples are illustrative and are not intended to be limiting. It should be understood that in some embodiments, an exemplary fitness function may be, for example, The number of input nodes, node layers, hidden layers, connections, computational complexity, etc. related to the error frequency or error rate The estimates may be based on factors other than the actual results.

[0234] In some embodiments, and optionally, any of the embodiments described above or below. In combination, the present disclosure utilizes at least one of the following aspects: This is also fine. U.S. Patent Application No. 8,195,484, Insurance product, ratin It is called the g system and method. U.S. Patent Application No. 8,548,833, Insurance product, ratin It is called the g system and method. U.S. Patent Application No. 8,554,588, Insurance product, ratin It is called the g system and method. U.S. Patent Application No. 8,554,589, Insurance product, ratin It is called the g system and method. U.S. Patent Application No. 8,595,036, Insurance product, ratin It is called the g system and method. U.S. Patent Application No. 8,676,610, Insurance product, ratin It is called the g system and method. U.S. Patent Application No. 8719059, Insurance product, ratin It is called the g system and method. U.S. Patent Application No. 8812331, Insurance product, ratin g and credit enhancement system and meth It has the name od for insuring project savings. .

[0235] At least some aspects of the present disclosure will now be described with reference to the following numbered paragraphs: do. [Item 1] At least one processor is configured to generate at least one user activity-related at least one receiving a training data set of a response variable representing another activity-related attribute; , and determining, by the at least one processor, one or more outliers. receiving at least one bias criterion for The at least one processor calculates model parameters for the machine learning model. determining a set, (1) by the at least one processor, generating a set of initial model parameters; 4. Applying the machine learning model to the training data set to obtain a set of model predictions. determining the (2) by said at least one processor, computing said set of model predictions on said train; The error of a data element is calculated by comparing it with the corresponding actual value in the training data set. generating a difference set; (3) determining by said at least one processor an error set of said data element errors and and identifying non-outlier response variables based at least in part on the at least one bias criterion. generating a data selection vector for distinguishing between the (4) performing, by the at least one processor, a process for generating a training data set; and utilizing the data selection vector to generate a non-outlier data set. (5) calculating, by the at least one processor, a classification result based on the non-outlier data set; determining a set of updated model parameters for the machine learning model using the (6) at least one abort execution termination by said at least one processor; Repeat steps (1) to (5) as iterations until a criterion is met. of the model parameters for the machine learning model as updated model parameters. and obtaining a set of the updated model parameters, whereby each iteration the set of predicted values ​​using the set of data as the set of initial model parameters, Regenerating the error set, the data selection vector, and the non-outlier data set. thing and The at least one processor detects the training data set and the A classifier of the outlier classifier machine learning model based at least in part on the data selection vector. Training a set of model parameters, said set including at least one outlier A trained outlier classifier machine learning model configured to identify value data elements. obtaining a The trained outlier classifier machine is a learning model coupled to a dataset of activity-related data relating to the at least one user activity; Applying to i) a set of outlier activity-related data in the dataset of activity-related data; and ii) a set of non-outlier activity-related data in the dataset of activity-related data. determining The at least one processor then computes the machine learning model based on the non-outlier activity function. a set of associated data elements to predict future activity associated with said at least one user activity; predicting dynamic related attributes; A method comprising: [Item 2] At least one computer-readable storage medium having software instructions stored thereon that communicates with the non-transitory computer-readable storage medium. A system including at least one processor, The software instructions, when executed, At least one of the at least one processors is configured to detect at least one user activity. receiving a training data set of a response variable representing at least one activity-related attribute; P and A step that receives at least one bias criterion that is used to determine one or more outliers. Top and Determining a set of model parameters for the machine learning model, (1) training the machine learning model with a set of initial model parameters; applying the model to the dataset to determine a set of model predictions; (2) comparing the set of model predicted values ​​with the corresponding actual values ​​in the training data set; generating an error set of data element errors by comparing; (3) determining whether said error set of data element errors and said at least one bias criterion are satisfied; Generate a data selection vector for identifying non-outlier target variables based in part on thing, (4) applying the data selection vector to the training data set to a non- generating an outlier dataset; (5) detecting an updated model for the machine learning model based on the non-outlier dataset; determining a set of model parameters; and (6) Step as iterations until at least one truncation run termination criterion is met. By repeating (1) to (5), the machine is updated as the model parameters. obtaining said set of model parameters for a learning model, whereby Each iteration then subtracts the set of updated model parameters from the set of initial model parameters. the set of predicted values, the set of errors, the data selection vector, and regenerating the non-outlier data set. and Based at least in part on the training data set and the data selection vector. We train a set of classifier model parameters for the outlier classifier machine learning model using a step of performing a training on the at least one outlier data element, the training being configured to identify at least one outlier data element; obtaining a trained outlier classifier machine learning model; The trained outlier classifier machine learning model is then fed to the at least one user. Applying it to a dataset of activity-related data on activities, i) a set of outlier activity-related data in the dataset of activity-related data; and ii) a set of non-outlier activity-related data in the dataset of activity-related data. determining applying the machine learning model to the set of non-outlier activity related data elements to predicting future activity-related attributes associated with at least one user activity; A system that executes the above. [Item 3] The data selection vector is fed to the training data by the at least one processor. applying the method to the training dataset to determine an outlier training dataset; and performing, by the at least one processor, a classification process using the outlier training data set. At least one outlier-specific machine learning model is used to training the model parameters to predict outlier data values; Utilizing the outlier-specific machine learning model, by the at least one processor, predicting an outlier activity-related data value for said set of outlier activity-related data. and 3. The system and method according to item 1 and / or 2, further comprising: [Item 4] by said at least one processor, using said training data set. ,Train the generalized model parameters of the generalized machine learning model to predicting a data value; utilizing the generalized machine learning model by the at least one processor. and predicting an outlier activity related data value for the set of outlier activity related data. Top and utilizing the generalized machine learning model by the at least one processor. predicting said activity-related data value; 3. The system and method according to item 1 and / or 2, further comprising: [Item 5] The data selection vector is fed to the training data by the at least one processor. applying the method to the training dataset to determine an outlier training dataset; and performing, by the at least one processor, a classification process using the outlier training data set. We use the,outlierspecific model parameters of the outlier-specific machine learning model to,train the outlier-specific model parameters of the outlier-specific predicting outlier data values; by said at least one processor, using said training data set. ,Train the generalized model parameters of the generalized machine learning model to predicting a data value; utilizing the outlier-specific machine learning model by the at least one processor. predicting an outlier activity related data value for said set of outlier activity related data. P and utilizing the outlier-specific machine learning model by the at least one processor. predicting said activity-related data value; 3. The system and method according to item 1 and / or 2, further comprising: [Item 6] by said at least one processor, using said training data set. ,Train the generalized model parameters of the generalized machine learning model to predicting a data value; utilizing the generalized machine learning model by the at least one processor. predicting the activity-related data value for the set of activity-related data using the Utilizing the outlier classifier machine learning model by the at least one processor. identifying outlier activity-related data values ​​from said activity-related data values; A step of removing the outlier activity related data values ​​by the at least one processor. Tep and 3. The system and method according to item 1 and / or 2, further comprising: [Item 7] The training data set is a function of concrete composition and concrete hardening evolution. Item 1, including the at least one activity-related attribute of concrete compressive strength as / or the system and method described in 2. [Item 8] The training data set includes energy as a function of home environmental conditions and lighting conditions. 3. The system according to item 1 and / or 2, comprising the at least one activity-related attribute of usage data. Stem and method. [Item 9] a prediction having at least one data element, Receives API (Application Programming Interface) requests to generate The step of believing, scheduling, by the at least one processor, execution of the machine learning model; The step of instantiating at least one cloud computing resource to P and According to the schedule executed by the at least one processor, Utilizing the machine learning model to generate at least one predicting activity-related data element values ​​of said at least one processor, returning a value to a computing device associated with the API request; 3. The system and method according to item 1 and / or 2, further comprising: [Item 10] The training data set may include at least one of three-dimensional patient images of a medical data set. also contains one activity-related attribute, The machine learning model comprises two or more physics-based lenses based on the medical dataset. Item 1 and configured to predict the activity-related data value including a staggering parameter. and / or the system and method according to claim 2. [Item 11] The training data set is a set of simulated control the at least one activity-related attribute of the results; The machine learning model is adapted to process the activity-related data including a control command for the electronic machine. 3. The system and method according to item 1 and / or 2, configured to predict a value. [Item 12] Compiling the set of activity-related data by the at least one processor into an activity-related data set. dividing the data into a plurality of subsets; The plurality of subsets of the activity-related data are then processed by the at least one processor. determining an ensemble model for each subset of activity-related data of And, The machine learning model comprises an ensemble of models; Each ensemble model is a random selection of models from the ensemble of models. and and, by the at least one processor, each ensemble model is individually predicting an ensemble specific activity related data value; The ensemble specific activity related data values ​​and determining an error in each ensemble model based on the known values; The at least one processor selects a best performing ensemble based on a minimum error. Steps to select a model 3. The system and method according to item 1 and / or 2, further comprising:

[0236] Although one or more embodiments of the present disclosure have been described, these embodiments are intended to be illustrative and not limiting. The present invention relates to a methodology, a system, and a method for the preparation of a pharmaceutical composition, a pharmaceutical The various embodiments of the system / platform and the apparatus of the present invention may be combined with each other in any combination. It will be appreciated that numerous modifications may be apparent to one skilled in the art, including those that may be utilized. Additionally, the various steps may be performed in any desired order (and may be implemented in any Any desired steps may be added and / or any desired steps may be removed. ).

Claims

1. receiving, by at least one processor, a training data set representative of at least one attribute related to at least one user activity; receiving, by the at least one processor, at least one bias criterion used to determine one or more outliers; determining, by the at least one processor, a set of model parameters for a machine learning model, the at least one processor comprising: (1) applying, by the at least one processor, the machine learning model having a set of initial model parameters to the training data set to determine a set of model predictions; (2) generating, by the at least one processor, an error set of data element errors by comparing the set of model predictions to corresponding actual values ​​of the training data set; (3) generating, by the at least one processor, a data selection vector for identifying non-outlier response variables based at least in part on the error set of data element errors and the at least one bias criterion; (4) utilizing, by the at least one processor, the data selection vector on the training data set to generate a non-outlier data set; and (5) determining, by the at least one processor, an updated set of model parameters for the machine learning model based on the non-outlier data set. and iteratively refining, by the at least one processor, the updated set of model parameters until a termination criterion is met, where the iteratively refining includes determining a plurality of model predictions using the updated set of model parameters and the set of model parameters based on the training data set; training, by the at least one processor, a set of classifier model parameters of an outlier classifier machine learning model based at least in part on the training data set and the data selection vector to obtain a trained outlier classifier machine learning model, the set of classifier model parameters being trained based at least in part on the training data set, the plurality of model predictions, and the data selection vector; outputting, by the at least one processor, the trained outlier classifier machine learning model including the set of classifier model parameters; A method comprising:

2. applying, by the at least one processor, the data selection vector to the training data set to determine an outlier training data set; training, by the at least one processor, at least one outlier-specific model parameter of at least one outlier-specific machine learning model using the outlier training data set to predict outlier data values; utilizing the outlier-specific machine learning model to predict, by the at least one processor, an outlier activity-related data value for a set of outlier activity-related data; The method of claim 1 further comprising:

3. training, by the at least one processor, generalized model parameters of a generalized machine learning model using the training data set to predict data values; utilizing the generalized machine learning model to predict, by the at least one processor, an outlier activity-related data value for a set of outlier activity-related data; utilizing the generalized machine learning model to predict, by the at least one processor, one or more activity-related data values; The method of claim 1 further comprising:

4. applying, by the at least one processor, the data selection vector to the training data set to determine an outlier training data set; training, by the at least one processor, outlier-specific model parameters of an outlier-specific machine learning model using the outlier training data set to predict outlier data values; training, by the at least one processor, generalized model parameters of a generalized machine learning model using the training data set to predict data values; predicting, by the at least one processor, an outlier-specific machine learning model for a set of outlier activity-related data; utilizing the generalized machine learning model to predict, by the at least one processor, one or more activity-related data values; The method of claim 1 further comprising:

5. training, by the at least one processor, generalized model parameters of a generalized machine learning model using the training data set to predict data values; utilizing the generalized machine learning model to predict, by the at least one processor, one or more activity-related data values ​​for a set of activity-related data; utilizing the outlier classifier machine learning model to identify, by the at least one processor, outlier activity-related data values ​​of the one or more activity-related data values; removing, by the at least one processor, the outlier activity related data values; The method of claim 1 further comprising:

6. The method of claim 1 , wherein the training data set includes at least one activity-related attribute of concrete compressive strength as a function of concrete composition and concrete hardening evolution.

7. The method of claim 1 , wherein the training data set includes at least one activity-related attribute of energy usage data as a function of home environmental conditions and lighting conditions.

8. receiving, by the at least one processor, an application programming interface (API) request to generate a prediction having at least one data element; instantiating, by the at least one processor, at least one cloud computing resource for scheduling execution of the machine learning model; utilizing the machine learning model to predict at least one activity-related data element value for the at least one data element according to the schedule executed by the at least one processor; returning, by the at least one processor, the at least one activity-related data element value to a computing device associated with the API request; The method of claim 1 further comprising:

9. the training data set includes at least one activity-related attribute of three-dimensional patient images of a medical data set; The method of claim 1 , wherein the machine learning model is configured to predict one or more activity-related data values ​​including two or more physically-based rendering parameters based on the medical dataset.

10. The training data set includes at least one activity-related attribute of a simulated control outcome for a command of an electronic machine; The method of claim 1 , wherein the machine learning model is configured to predict one or more activity-related data values ​​including control commands for the electronic machine.

11. dividing, by the at least one processor, the set of activity-related data into a plurality of subsets of activity-related data; determining, by the at least one processor, an ensemble model for each subset of activity-related data among the subsets of activity-related data, the machine learning model comprising an ensemble of models, each ensemble model comprising a random combination of models from the ensemble of models; predicting, by the at least one processor, ensemble-specific activity-related data values ​​individually using each ensemble model; determining, by the at least one processor, an error in each ensemble model based on the ensemble specific activity-related data values ​​and known values; selecting, by the at least one processor, a best performing ensemble model based on a minimum error; The method of claim 1 further comprising:

12. 1. A system comprising at least one processor in communication with a non-transitory computer readable storage medium having software instructions stored thereon, The software instructions, when executed, cause the at least one processor to: receiving a training data set representative of at least one attribute related to at least one user activity; receiving at least one bias criterion used to determine one or more outliers; determining a set of initial model parameters for a machine learning model, comprising: (1) applying the machine learning model having the set of initial model parameters to the training dataset to determine a set of model predictions; (2) generating an error set of data element errors by comparing the set of model predictions to corresponding actual values ​​of the training dataset; (3) generating a data selection vector for identifying non-outlier response variables based at least in part on the error set of data element errors and the at least one bias criterion; (4) utilizing the data selection vector for the training dataset to generate a non-outlier dataset; and (5) determining an updated set of model parameters for the machine learning model based on the non-outlier dataset. and iteratively refining the updated set of model parameters until a termination criterion is met, where the iteratively refining includes determining a plurality of model predictions using the updated set of model parameters and the set of model parameters based on the training data set; training a set of classifier model parameters of an outlier classifier machine learning model to obtain a trained outlier classifier machine learning model, the set of classifier model parameters being trained based at least in part on the training data set, the plurality of model predictions, and the data selection vector; outputting the trained outlier classifier machine learning model including the set of classifier model parameters; A system that executes the above.

13. The software instructions, when executed, cause the at least one processor to: applying the data selection vector to the training data set to determine an outlier training data set; training at least one outlier-specific model parameter of at least one outlier-specific machine learning model using the outlier training data set to predict outlier data values; utilizing the outlier-specific machine learning model to predict outlier activity-related data values ​​for a set of outlier activity-related data; The system of claim 12 , further comprising:

14. The software instructions, when executed, cause the at least one processor to: training generalized model parameters of a generalized machine learning model using the training data set to predict data values; utilizing the generalized machine learning model to predict outlier activity-related data values ​​for a set of outlier activity-related data; utilizing the generalized machine learning model to predict one or more activity-related data values; The system of claim 12 , further comprising:

15. The software instructions, when executed, cause the at least one processor to: applying the data selection vector to the training data set to determine an outlier training data set; using the outlier training data set to train outlier-specific model parameters of an outlier-specific machine learning model to predict outlier data values; training generalized model parameters of a generalized machine learning model using the training data set to predict data values; utilizing the outlier-specific machine learning model to predict outlier activity-related data values ​​for a set of outlier activity-related data; utilizing the generalized machine learning model to predict one or more activity-related data values; The system of claim 12 , further comprising:

16. The software instructions, when executed, cause the at least one processor to: training generalized model parameters of a generalized machine learning model using the training data set to predict data values; utilizing the generalized machine learning model to predict one or more activity-related data values ​​for a set of activity-related data; utilizing the outlier classifier machine learning model to identify outlier activity-related data values ​​of the one or more activity-related data values; removing the outlier activity related data values; The system of claim 12 , further comprising:

17. The system of claim 12 , wherein the training data set includes at least one activity-related attribute of concrete compressive strength as a function of concrete composition and concrete hardening evolution.

18. The system of claim 12 , wherein the training data set includes at least one activity-related attribute of energy usage data as a function of home environmental conditions and lighting conditions.

19. The software instructions, when executed, cause the at least one processor to: receiving an application programming interface (API) request to generate a prediction having at least one data element; instantiating at least one cloud computing resource for scheduling execution of the machine learning model; utilizing the machine learning model to predict at least one activity-related data element value for the at least one data element according to the schedule that is executed; returning the at least one activity-related data element value to a computing device associated with the API request; The system of claim 12 , further comprising:

20. The software instructions, when executed, cause the at least one processor to: Dividing the set of activity-related data into a plurality of subsets of activity-related data; determining an ensemble model for each subset of activity-related data among the subsets of activity-related data, the machine learning model comprising an ensemble of models, each ensemble model comprising a random combination of models from the ensemble of models; utilizing each ensemble model to predict an ensemble-specific activity-related data value separately; determining an error in each ensemble model based on the ensemble specific activity related data values ​​and known values; selecting the best performing ensemble model based on the minimum error; The system of claim 12 , further comprising:

Images