Computer implementation methods, systems, and computer program products (decisions regarding model updates)

Abstract

To provide a computer-implemented method, a system and a computer program product for model update determination.SOLUTION: According to embodiments, a method, a device and a computer program product for model update determination are proposed. In the method, multiple historical data items and multiple new data items are obtained. The multiple historical data items are used for training a model, and the multiple new data items are applied to the model. At least one of an overall difference, a structural difference and a confidence difference between the multiple historical data items and the multiple new data items is determined. Thereby, an indication of whether to update the model is determined based on the at least one of the overall difference, the structural difference and the confidence difference.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to model management, and more particularly, to a method, a system, and a computer program product for model update determination.

Background Art

[0002] With the development of information technology, the use of various models such as causal models, machine learning models, neural network models, data analysis models, linear models, and non-linear models has been progressing. These models are useful for users to solve various problems.

Summary of the Invention

Problems to be Solved by the Invention

[0003] To provide a computer-implemented method, a system, and a computer program product for model update determination.

Means for Solving the Problems

[0004] According to one embodiment, a computer-implemented method is provided. According to the method, a plurality of historical data items and a plurality of new data items are obtained. The plurality of historical data items are used for training a model, and the plurality of new data items are applied to the model. At least one of an overall difference, a structural difference, and a reliability difference between the plurality of historical data items and the plurality of new data items is determined. Thereby, an instruction on whether to update the model is determined based on the at least one of the overall difference, the structural difference, and the reliability difference.

[0005] According to one embodiment, a system is provided. The system includes a processing unit and a memory coupled to the processing unit that stores instructions therein. When the instruction is executed by the processing unit, it performs a method which includes: obtaining a plurality of historical data items and a plurality of new data items, wherein the plurality of historical data items are used to train a model and the plurality of new data items are applied to the model; determining at least one of a total difference, a structural difference, and a confidence difference between the plurality of historical data items and the plurality of new data items; and determining an instruction whether to update the model based on at least one of the total difference, the structural difference, and the confidence difference.

[0006] According to one embodiment, a computer program product is provided which includes a computer-readable storage medium in which program instructions are implemented. The program instructions are executable by a processor and cause the processor to perform an operation which includes acquiring a plurality of historical data items and a plurality of new data items, wherein the plurality of historical data items are used to train a model and the plurality of new data items are applied to the model; determining at least one of the total difference, the structural difference, and the confidence difference between the plurality of historical data items and the plurality of new data items; and determining an instruction whether to update the model based on the at least one of the total difference, the structural difference, and the confidence difference.

[0007] Through a more detailed description of some embodiments of the present disclosure in the accompanying drawings, the above and other objects, features and advantages of the present disclosure will become more apparent, where the same references generally refer to the same components in embodiments of the present disclosure. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows a cloud computing node according to one embodiment. [Figure 2]Figure 2 shows a cloud computing environment according to one embodiment. [Figure 3] Figure 3 shows an abstraction model layer according to one embodiment. [Figure 4] Figure 4 is a flowchart showing an example of a model management method according to one embodiment. [Figure 5] Figure 5 is a schematic diagram showing an example of a history data item according to one embodiment. [Figure 6] Figure 6 is a schematic diagram showing an example of a new data item according to one embodiment. [Figure 7] Figure 7 is a schematic diagram showing an example of clustering of historical data items according to one embodiment. [Figure 8] Figure 8 is a schematic diagram showing an example of clustering of new data items according to one embodiment. [Figure 9] Figure 9 is a schematic diagram showing an example of a target value distribution according to one embodiment. [Figure 10] Figure 10 is a schematic diagram showing an example of a confidence interval according to one embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, several embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings illustrating embodiments of the present disclosure. However, the present disclosure can be implemented in various ways and should not be construed as being limited to the embodiments disclosed herein.

[0010] While this disclosure includes a detailed description of cloud computing, the implementations of the teachings described herein are not limited to cloud computing environments. Rather, these embodiments can be implemented in any other type of computer environment that is currently known or may be developed in the future.

[0011] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0012] The characteristics are as follows:

[0013] On-demand self-service: Cloud consumers can unilaterally prepare computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0014] Broad network access: Computing power is available over the network and accessible through standard mechanisms. This facilitates utilization by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, PDAs).

[0015] Resource pooling: A provider's computing resources are pooled and delivered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated as needed. Generally, consumers have a sense of location independence because they do not manage or know the exact location of the resources provided. However, consumers may be able to identify the location at a higher level of abstraction (e.g., country, state, data center).

[0016] Rapid Elasticity: Computing capabilities can be prepared quickly and flexibly, so in some cases they can automatically scale out immediately and be released promptly to scale in immediately. To the consumer, the computing capabilities available for preparation often seem unlimited and can be purchased in any quantity at any time.

[0017] Measured Services: Cloud systems utilize measurement functions at a certain level of abstraction suitable for service types (e.g., storage, processing, bandwidth, active user count) to automatically control and optimize resource usage. It is possible to monitor, control, and report resource usage amounts to provide transparency to both the provider and the consumer of the services being utilized.

[0018] The service model is as follows.

[0019] Software as a Service (SaaS): The functions provided to the consumer are that they can utilize the provider's applications operating on the cloud infrastructure. The applications can be accessed from various client devices via a client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, server, operating system, storage, and even individual application functions. However, this does not apply to limited settings of application configurations specific to the user.

[0020] Platform as a Service (PaaS): The function provided to the consumer is to deploy the applications created or obtained by the consumer onto the cloud infrastructure by using the programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, and storage, but can control the deployed applications and, in some cases, also configure the hosting environment.

[0021] Infrastructure as a Service (IaaS): The function provided to the consumer is to prepare the processors, storage, network, and other basic computing resources on which the consumer can deploy and run any software including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but can control the operating systems, storage, and deployed applications and, in some cases, also partially control some network components (such as the host firewall).

[0022] The deployment models are as follows.

[0023] Private cloud: This cloud infrastructure is operated exclusively for a specific organization. This cloud infrastructure can be managed by the organization or a third party and can exist on-premises or off-premises.

[0024] Community cloud: This cloud infrastructure is shared by multiple organizations and supports a specific community with common concerns (such as missions, security requirements, policies, and compliance). This cloud infrastructure can be managed by the organization or a third party and can exist on-premises or off-premises.

[0025] Public Cloud: This cloud infrastructure is provided to a large number of people or large industry groups and is owned by organizations that sell cloud services.

[0026] Hybrid Cloud: This cloud infrastructure combines two or more cloud models (private, community, or public). While maintaining the unique entities of each model, they are bound together by standards or individual technologies to achieve data and application portability (e.g., cloud bursting for load balancing across clouds).

[0027] Cloud computing environments are service-oriented environments that emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure, which includes a network of interconnected nodes.

[0028] Figure 1 schematically shows an example of a cloud computing node. Note that cloud computing node 10 is merely one example of a suitable cloud computing node and does not imply any limitation on the use or scope of functionality of the embodiments described herein. In any case, cloud computing node 10 can implement, perform any of the functions described above, or both.

[0029] Cloud computing nodes 10 include portable devices such as computer systems / servers 12 or communication devices. These can operate with a number of other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, configurations, or combinations suitable for use with computer systems / servers 12 include personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0030] The computer system / server 12 can be described in general terms in relation to computer system executable instructions, such as program modules executed by the computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. The computer system / server 12 can be implemented in a distributed cloud computing environment where tasks are executed by remote processing units linked via a communication network. In a distributed cloud computing environment, program modules can be stored in both local and remote computer system storage media, including memory storage devices.

[0031] As shown in Figure 1, the computer system / server 12 in the cloud computing node 10 is shown as a general-purpose computer device. Examples of components of the computer system / server 12 include one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components, including the system memory 28, to the processor 16.

[0032] Bus 18 represents one or more of several types of bus structures, including memory buses or memory controllers using various bus architectures, peripheral buses, accelerated graphics ports (AGP), and processor or local buses. Examples of such architectures include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Expansion ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0033] The computer system / server 12 generally includes various computer system-readable media. Such media may be any available media accessible by the computer system / server 12, and may include both volatile and non-volatile media, as well as both removable and non-removable media.

[0034] The system memory 28 may include computer system-readable media as volatile memory, such as RAM 30, cache memory 32, or both. The computer system / server 12 may further include other removable / non-removable computer system-readable media and volatile / non-volatile computer system-readable media. As an example, the storage system 34 may be provided for reading and writing to a non-removable non-volatile magnetic medium (not shown; commonly referred to as a “hard drive”). Also, although not shown, a magnetic disk drive for reading and writing to removable non-volatile magnetic disks (e.g., floppy disks) and an optical disk drive for reading and writing to removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these examples, each may be connected to the bus 18 by one or more data medium interfaces. As further illustrated and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of this embodiment.

[0035] As an example, a program / utility 40 having a set (at least one) of program modules 42 can be stored in memory 28, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or several combinations thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions or methods, or both, of the embodiments described herein.

[0036] Furthermore, the computer system / server 12 can communicate with one or more external devices 14 such as a keyboard, pointing device, or display 24, one or more devices that enable interaction between the user and the computer system / server 12, or any device that enables communication between the computer system / server 12 and one or more other computer devices (e.g., a network card or modem), or a combination thereof. Such communication can be performed via the input / output (I / O) interface 22. In addition, the computer system / server 12 can communicate with one or more networks (such as a local area network (LAN), a general-purpose wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof) via the network adapter 20. As shown in the figure, the network adapter 20 communicates with other components of the computer system / server 12 via the bus 18. Although not shown in the figure, other hardware components, software components, or both can be used in conjunction with the computer system / server 12. Examples of these include microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0037] Figure 2 shows an exemplary cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10. Local computer devices used by cloud consumers (e.g., PDAs or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof) can communicate with these nodes. The nodes 10 can communicate with each other. The nodes 10 can be grouped physically or virtually (not shown) in one or more networks, such as the private, community, public, or hybrid clouds or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software as a service, or a combination thereof, without requiring cloud consumers to maintain resources on their local computer devices. Note that the types of computer devices 54A-N shown in Figure 2 are merely examples, and it should be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network addressable connection (e.g., using a web browser) or both.

[0038] Next, Figure 3 shows the set of functional abstraction layers provided by the cloud computing environment 50 (Figure 2). It should be understood that the components, layers, and functions shown in Figure 3 are illustrative and not limiting to the embodiments. As illustrated, the following layers and corresponding functions are provided.

[0039] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include a mainframe 61, a reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage 65, and a network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0040] The virtualization layer 70 provides an abstraction layer. From this layer, virtual entities such as virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75 can be provided.

[0041] As an example, the management layer 80 can provide the following functions: Resource preparation 81 enables the dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. As an example, these resources may include licenses for application software. Security enables not only protection of data and other resources but also identification and verification of cloud consumers and tasks. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables the allocation and management of cloud computing resources to ensure that requested service levels are met. Service Level Agreement (SLA) planning and execution 85 enables the pre-arrangement and procurement of cloud computing resources that are expected to be needed in the future in accordance with the SLA.

[0042] Workload layer 90 provides examples of capabilities available in a cloud computing environment. Examples of workloads and capabilities available from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and model management 96.

[0043] In some cases, a model may function well in its initial stages, built with historical data items. However, as new data items are added, the model may degrade over time. In this case, it may be desirable to decide whether to update the model. However, traditional model update decisions can be inefficient and inaccurate.

[0044] Traditionally, if target values ​​can be obtained for new data items, the performance of the original model can be determined using those target values. However, sometimes obtaining target values ​​is difficult, or only predicted values ​​for the new data items can be obtained.

[0045] Because target values ​​may not always be accessible, various methods have been proposed to determine whether to update the original model. For example, traditional methods compare the variable distributions between new and historical data items individually. For categorical variables, a chi-squared test is used to examine the categorical value distribution between new and historical data items. For continuous variables, the range of the continuous variable is divided into many intervals, and a chi-squared test is used to check the values ​​in each interval between new and historical data items. The results of the chi-squared tests for all variables are then averaged to obtain the final result. However, such traditional methods cannot account for various combinations of factors and can be inefficient, inaccurate, or both.

[0046] An improved solution for model management is provided in this embodiment. Generally, according to the embodiments of this disclosure, a plurality of historical data items and a plurality of new data items are acquired. The plurality of historical data items are used to train the model, and the plurality of new data items are applied to the model. At least one of the overall difference, structural difference, and confidence difference is determined between the plurality of historical data items and the plurality of new data items. Thereafter, an instruction to update the model is determined based on at least one of the overall difference, structural difference, and confidence difference.

[0047] According to the model management proposed herein, the overall differences, structural differences, or confidence differences, or a combination thereof, between multiple historical data items and multiple new data items can be used in combination to achieve efficient and accurate model update decisions.

[0048] Here, several exemplary embodiments will be described with reference to Figures 4 to 8. Figure 4 is a flowchart of an example of a model management method 400 according to one embodiment. The model management method 400 can be implemented at least partially by a computer system / server 12 or other suitable system.

[0049] In operation 410, the computer system / server 12 retrieves multiple historical data items and multiple new data items. The multiple historical data items are used to train the model, and the multiple new data items are applied to the model.

[0050] Figure 5 is a schematic diagram 500 showing an example of a history data item according to one embodiment. Figure 5 shows a plurality of history data items 510-1 to 510-N (hereinafter collectively referred to as "history data item 510," where N is an integer greater than 1). Each of the history data items 510 may include at least one field such as gender, date of birth, salary, and working hours. It should be understood that the history data items 510 may include fields other than those shown in Figure 5.

[0051] Historical data item 510 was used to train or create models for solving various problems and performing various analyses. The models could be any suitable model, such as causal models, machine learning models, neural network models, data analysis models, linear models, or nonlinear models.

[0052] As mentioned above, the model works well in the initial stages when it is trained or built on historical data item 510. However, as new data items are added, the model degrades over time because new data items may differ from historical data item 510.

[0053] Figure 6 is a schematic diagram 600 showing an example of a new data item according to one embodiment. Figure 6 shows a plurality of new data items 610-1 to 610-M (hereinafter collectively referred to as "new data item 610", where M is an integer greater than 1). Similar to the historical data item 510, each of the new data items 610 may include at least one field such as gender, date of birth, salary, and working hours.

[0054] Although the fields included in the new data item 610 are shown as being the same as those included in the historical data item 510, it should be understood that the fields in the new data item 610 and the historical data item 510 may be different. Furthermore, the new data item 610 and the historical data item 510 are merely examples, and the contents of these data items can be any appropriate content.

[0055] Returning to Figure 4, in operation 420, the computer system / server 12 determines at least one of the overall difference, structural difference, and confidence difference between the multiple historical data items 510 and the multiple new data items 610. Based on this, in operation 430, the computer system / server 12 determines whether to update the model based on at least one of the overall difference, structural difference, and confidence difference.

[0056] For example, the computer system / server 12 may obtain weights for the overall difference, structural difference, and confidence difference, respectively. In some embodiments, the weights may be initially set by the user and learned during the model update process.

[0057] Next, the computer system / server 12 can weight the overall difference, structural difference, and confidence difference according to their respective weights, and use the sum of the weighted differences to determine whether to update the model. For example, the computer system / server 12 may compare the sum to a predetermined threshold. If the sum is greater than the predetermined threshold, i.e., if the difference between the new data item 610 and the historical data item 510 is significant, the instruction may indicate that the model should be updated.

[0058] By considering the overall difference, structural difference, or confidence difference, or a combination thereof, between the historical data item 510 and the new data item 610, efficient and accurate model update decisions can be achieved.

[0059] The following text will explain in detail how to determine the overall difference, structural difference, and confidence difference.

[0060] The overall difference is associated with the proportion of new data items 610 that are dissimilar to the historical data items 510 for all new data items 610. In some embodiments, to determine the overall difference, the computer system / server 12 may cluster multiple historical data items 510 into multiple clusters (hereinafter referred to as "first clusters"). In this case, similar historical data items 510 may be clustered into similar clusters. In some embodiments, the historical data items 510 may be clustered using any suitable clustering technique, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise) or K-means.

[0061] Figure 7 is a schematic diagram 700 showing an example of clustering of historical data items according to one embodiment. As shown in Figure 7, the historical data items 510 are clustered into three clusters, namely clusters 1 to 3.

[0062] Furthermore, in some embodiments, the computer system / server 12 may cluster multiple new data items 610 into multiple clusters (hereinafter referred to as "second clusters"). In this case, similar new data items 610 may be clustered into similar clusters. Similar to the historical data items 510, the new data items 610 may be clustered using any suitable clustering technique.

[0063] In some embodiments, the new data item 610 and the historical data item 510 may be clustered using similar clustering techniques. For example, both the new data item 610 and the historical data item 510 may be clustered using DBSCAN. In this case, the new data item and the historical data item clustered in the same cluster can be treated as similar data items. For example, suppose that the historical data items 510-2, 510-5, and 510-9 are clustered in cluster 3, and the new data item 610-2 is also clustered in cluster 3. In this case, the historical data items 510-2, 510-5, and 510-9 and the new data item 610-2 can be treated as similar data items.

[0064] Figure 8 is a schematic diagram 800 showing an example of clustering of new data items according to one embodiment. As shown in Figure 8, the new data items 610 are clustered into four clusters. Specifically, new data item 610-1 is clustered into cluster 1, new data item 610-2 into cluster 3, new data item 610-3 into cluster 2, new data item 610-4 into cluster 1, and new data item 610-M into cluster 1.

[0065] Regarding cluster-1, as mentioned above, new data items 610 may differ from historical data items 510. In this case, some new data items may be clustered into a different cluster than the cluster of historical data items 510. Such a different cluster may be denoted as cluster-1 to indicate that new data item 610-4 is an outlier new data item that does not resemble any of the historical data items 510.

[0066] Furthermore, as described above, the overall difference is associated with the proportion of new data items that are dissimilar to the historical data items 510 in all new data items 610. In this case, to determine the overall difference, the computer system / server 12 may determine the first number of data items in multiple new data items 610, i.e., the total number of all new data items in multiple new data items 610.

[0067] Furthermore, the computer system / server 12 may also select outlier new data items from a group of new data items. The cluster of an outlier new data item is different from the first group of clusters. For example, new data item 610-4 is an outlier new data item, and cluster-1 of new data item 610-4 is different from cluster-1-3. For this reason, the computer system / server 12 may select a first set of new data items from a group of new data items. The cluster of each data item in the first set of new data items, according to the second group of clusters, is different from the first group of clusters. In this case, the computer system / server 12 may determine a second number of data items in the first set of data items, i.e., the total number of all data items in the first set of data items.

[0068] The computer system / server 12 may then determine the overall difference based on the first and second numbers. For example, the overall difference may be the quotient obtained by dividing the second number by the first number. In this case, the more outlier new data items there are, the higher the overall difference will be, which means that the new data items 610 are more different from the historical data items 510.

[0069] The determination of the overall difference was as described above, but the following text will further explain the structural difference.

[0070] The structural difference relates to the structural difference between the new data items 610 and the historical data items 510. Specifically, the structural difference relates to the proportion of new data items 610 that have a structure dissimilar to the historical data items 510. In some embodiments, to determine the structural difference, the computer system / server 12 may select a set of new data items from all new data items 610 that are similar to the historical data items 510. In this case, outlier new data items are excluded from the selection.

[0071] Specifically, the computer system / server 12 may select a second set of new data items from a plurality of new data items. The cluster of each data item in the second set of new data items, according to the second plurality of clusters, is the same as the cluster of the first plurality of clusters. For example, new data items 610-1, 610-2, 610-3, and 610-M may be selected for the second set of new data items because their clusters belong to clusters 1-3 of the historical data item 510.

[0072] Next, the computer system / server 12 may select a third set of new data items from a second set of new data items. The structure of each data item in the third set of new data items is different from the structure of multiple historical data items. In this case, the computer system / server 12 may specify a third number of data items in the third set of new data items, that is, the total number of data items in the third set of new data items.

[0073] As described above, the structural difference is associated with the proportion of new data items 610 that have a structure dissimilar to that of historical data items 510. In this case, the computer system / server 12 may determine the structural difference based on the first and third numbers of data items in multiple new data items. For example, the structural difference may be the quotient obtained by dividing the third number by the first number. In this case, the more new data items that have a structure different from that of historical data items 510, the larger the structural difference will be, which also means that the new data items 610 are more different from the historical data items 510.

[0074] In some embodiments, in order to select a third set of new data items, the computer system / server 12 may perform the following processing on each data item of the second set of new data items. For example, the computer system / server 12 may select historical data items in a cluster similar to that of the new data items. Specifically, the computer system / server 12 may determine the cluster of the new data items. Furthermore, the computer system / server 12 may select a first set of historical data items from a plurality of historical data items 510. The first set of historical data items is clustered in a cluster similar to that of the new data items.

[0075] For example, suppose new data item 610-2 and historical data items 510-2, 510-5, and 510-9 are clustered in cluster 3. In this case, historical data items 510-2, 510-5, and 510-9 are selected for new data item 610-2.

[0076] Typically, the first set of new and historical data items is multidimensional. In this case, to compare the structure of the first set of new and historical data items, these data items can be projected from multiple dimensions into a single dimension. For example, as mentioned above, each data item can contain multiple fields, and each field can be treated as one dimension. Therefore, each data item can contain multiple dimensions.

[0077] To project a data item, the computer system / server 12 may determine at least one set of weights, such as [a1, a2, a3, ..., ai], [b1, b2, b3, ..., bi], [c1, c2, c3, ..., ci], where i represents the number of dimensions or fields of the data item. In some embodiments, the set of weights may be determined randomly. Alternatively or additionally, higher weights may be assigned to more important dimensions or fields. For example, if the gender field is more important than other fields, the weight of the gender field may be higher than that of the other fields. Furthermore, in some embodiments, the set of weights may be normalized such that the sum of all weights in the set equals 1. For example, for the set of weights [a1, a2, a3, ..., ai], the sum of a1, a2, a3, ..., ai is 1.

[0078] Data items can be weighted by a set of weights to transform multidimensional data items into a one-dimensional projection result. Specifically, the value of each field of a data item can be multiplied by the corresponding weight in the set of weights. The projection result can be called the target value. The target values ​​of the first set of historical data items can form a target value distribution.

[0079] Furthermore, it can be seen that each set of weights can derive a projection result. In this case, if there is at least one set of weights, a new data item can be projected onto at least one target value. Each of the at least one target value corresponds to each set of weights in at least one set of weights. Similarly, using at least one set of weights, a first set of historical data items can be projected onto at least one target value distribution. Each of the at least one target value distribution corresponds to each set of weights in at least one set of weights.

[0080] Figure 9 is a schematic diagram 900 showing an example of a target value distribution according to one embodiment. As shown in Figure 9, a set of historical data items can be projected onto three target value distributions 910 to 930, each having a set of three weights.

[0081] Next, the computer system / server 12 may check whether the target value of the new data item falls within a predetermined confidence interval (e.g., a 90% confidence interval or any appropriate confidence interval) of the corresponding target value distribution of the first set of historical data items. Figure 9 shows similar confidence intervals for the three target value distributions 910-930, but it should be understood that the target value distributions 910-930 may have different confidence intervals.

[0082] If a target value falls within a predetermined confidence interval, it means that the target value belongs to the target value distribution. For example, if we assume that the target value of a new data item projected from a set of weights [a1, a2, a3, ..., ai] is within the 90% confidence interval of the corresponding target value distribution of the first set of historical data items projected from a similar set of weights [a1, a2, a3, ..., ai], then the target value belongs to the target value distribution. In this case, the new data item has a similar structure to the first set of historical data items.

[0083] On the other hand, if at least one of the target values ​​is not all within its corresponding target value distribution, it means that the new data item has a different structure from the first set of historical data items. For example, if all three target values ​​of a new data item projected from three sets of weights [a1, a2, a3, ..., ai], [b1, b2, b3, ..., bi], [c1, c2, c3, ..., ci] are not within the respective 90% confidence intervals of the target value distributions of the first set of historical data items projected from similar sets of weights [a1, a2, a3, ..., ai], [b1, b2, b3, ..., bi], [c1, c2, c3, ..., ci], then the new data item has a different structure from the first set of historical data items. The first set of historical data items consists of the data items that are most similar to the new data item among all the historical data items 510. Therefore, if the new data item has a different structure from the first set of historical data items, it must have a different structure from all the historical data items 510. In this way, the new data item can be determined to be a data item in the third set of new data items.

[0084] While the determination of structural differences was mentioned earlier, the following text will further explain the difference in confidence levels.

[0085] The confidence difference relates to the difference in confidence or predicted values ​​obtained by applying the new data items 610 and the historical data items 510 to the model, respectively. Specifically, the confidence difference relates to the proportion of new data items 610 that have a confidence level dissimilar to the historical data items 510. In some embodiments, to determine the confidence difference, the computer system / server 12 may select a set of new data items from all new data items 610 that are similar to the historical data items 510. In this case, outlier new data items are excluded from the selection.

[0086] Specifically, the computer system / server 12 may select a fourth set of new data items from a plurality of new data items. The cluster of each data item in the fourth set of new data items, according to the second plurality of clusters, is the same as the cluster of the first plurality of clusters. For example, new data items 610-1, 610-2, 610-3, and 610-M may be selected for the set of new data items because their clusters belong to clusters 1-3 of the historical data item 510. It should be understood that the fourth set of new data items may be the same as the second set of new data items. In this case, the second set of new data items may be used directly as the fourth set of new data items, and the selection of the fourth set of new data items may be omitted.

[0087] Next, the computer system / server 12 may select a fifth set of new data items from the fourth set of new data items. The confidence level of each data item in the fifth set of new data items is different from the confidence levels of multiple historical data items. In this case, the computer system / server 12 may determine the number of data items in the fourth set of new data items, that is, the total number of data items in the fifth set of new data items.

[0088] As described above, the confidence difference is associated with the proportion of new data items 610 that have a confidence level that is dissimilar to that of the historical data item 510. In this case, the computer system / server 12 may determine the confidence difference based on the first and fourth numbers of items in multiple new data items. For example, the confidence difference may be the quotient obtained by dividing the fourth number by the first number. In this case, the more new data items that have a confidence level different from that of the historical data item 510, the larger the confidence difference will be, which also means that the new data item 610 is more different from the historical data item 510.

[0089] In some embodiments, in order to select a fifth set of new data items, the computer system / server 12 may perform the following processing on each data item of the fourth set of new data items. For example, the computer system / server 12 may select historical data items in a cluster similar to that of the new data items. Specifically, the computer system / server 12 may determine the cluster of the new data items. Next, the computer system / server 12 may select a second set of historical data items from a plurality of historical data items. The second set of historical data items is clustered into clusters.

[0090] For example, suppose new data item 610-2 and historical data items 510-2, 510-5, and 510-9 are clustered in cluster 3. In this case, historical data items 510-2, 510-5, and 510-9 are selected for new data item 610-2.

[0091] Next, the computer system / server 12 may determine the confidence level of the new data item and the confidence interval for a second set of historical data items. To determine the confidence level of the new data item, the computer system / server 12 may apply the new data item to the model.

[0092] Furthermore, in order to determine the confidence intervals for a second set of historical data items, the computer system / server 12 may apply the second set of historical data items to the model to obtain a set of intermediate confidence intervals. Specifically, for each of the second set of historical data items, the model may output a predetermined intermediate confidence interval (such as a 95% confidence interval or any appropriate confidence interval). Each intermediate confidence interval may have a lower limit and an upper limit.

[0093] Furthermore, the confidence interval may be determined from a set of intermediate confidence intervals. For example, the set of intermediate confidence intervals may be averaged to obtain a confidence interval. Alternatively, the smallest lower bound of the set of intermediate confidence intervals may be used as the lower bound of the confidence interval, and the largest upper bound of the set of intermediate confidence intervals may be used as the upper bound of the confidence interval.

[0094] Figure 10 is a schematic diagram 1000 showing an example of a confidence interval according to one embodiment. As shown in Figure 10, the set of intermediate confidence intervals may be averaged to obtain a confidence interval. Specifically, the lower and upper limits of the set of intermediate confidence intervals are averaged to obtain the lower and upper limits of the confidence interval. If the confidence level of a new data item does not fall within the confidence interval, it means that the new data item has a different confidence level from the second set of historical data items. Since the historical data items are the data items most similar to the new data item, if the new data item has a different confidence level from the second set of historical data items, then the new data item should have a different confidence level from all the historical data items 510.

[0095] Furthermore, in some embodiments, the computer system / server 12 may tune the model to improve the accuracy of the confidence intervals. For example, the regression coefficients of a linear model may be tuned. In this case, the computer system / server 12 may tune the model parameters to obtain at least one tuned model, and may apply new data items to at least one tuned model to determine at least one confidence level. Each of the at least one confidence level corresponds to each of the at least one tuned models.

[0096] Furthermore, the computer system / server 12 may apply a second set of historical data items to at least one adjusted model in order to determine at least one confidence interval. Each of the at least one confidence intervals corresponds to each of the at least one adjusted model.

[0097] In this case, if the confidence level of a data item does not fall within the confidence interval of the second set of historical data items, the computer system / server 12 may determine that the data item is a data item in the fifth set of new data items. Furthermore, in some embodiments, the model is adjusted to obtain at least one adjusted model. In this case, if all at least one confidence levels do not fall within the confidence interval of each of the at least one confidence intervals, the computer system / server 12 may determine that the data item is a data item in the fifth set of new data items.

[0098] In this way, the overall difference, structural difference, or confidence difference, or a combination thereof, between the historical data item 510 and the new data item 610 is appropriately determined, thereby enabling efficient and accurate model update decisions based on these differences.

[0099] This embodiment may be a system, method, or computer program product or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium storing computer-readable program instructions for causing a processor to execute an aspect of this embodiment.

[0100] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. Examples of computer-readable storage media may be electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or appropriate combinations thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROM, DVD, memory stick, floppy disk, punch cards, or grooved raised structures, and mechanically encoded devices on which instructions are recorded, and appropriate combinations thereof. Computer-readable storage devices as used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0101] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer device / processor. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, LAN, WAN, or wireless network, or a combination thereof). The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface within each computer device / processor receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium in the respective computer device / processor.

[0102] The computer-readable program instructions for performing the operation of this embodiment may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions can be executed as a standalone software package, either entirely on the user's computer or partially on the user's computer. Alternatively, they can be executed partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including LANs and WANs, or it may be connected to an external computer (for example, via the Internet using an Internet Service Provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), and programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of computer-readable program instructions in order to customize the electronic circuits for the purpose of performing the aspects of this embodiment.

[0103] Each aspect of this embodiment is described herein with reference to flowcharts or block diagrams, or both, of the methods, apparatus (systems), and computer program products of the embodiment. Each block in a flowchart or block diagram, or both, and combinations of multiple blocks in a flowchart or block diagram, or both, can be executed by computer-readable program instructions.

[0104] The above computer-readable program instructions may be provided to a processor of a computer or other programmable data processing device for the purpose of producing a machine. This allows these instructions, executed via such computer or other programmable data processing device processor, to form means for performing functions / operations identified in one or more blocks in a flowchart or block diagram, or both. The above computer-readable program instructions may further be stored in a computer-readable storage medium that can be instructed to function in a particular manner to a computer, programmable data processing device, or other device, or a combination thereof. This allows the computer-readable storage medium containing the instructions to constitute a product containing instructions that perform functions / operations identified in one or more blocks in a flowchart or block diagram, or both.

[0105] Alternatively, a computer execution process may be generated by loading computer-readable program instructions into a computer, another programmable device, or other device, and executing a series of operational steps on the computer, other programmable device, or other device. This ensures that the instructions executed on the computer, other programmable device, or other device perform functions / operations identified by one or more blocks in a flowchart, block diagram, or both.

[0106] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for performing a particular logical function. In some other implementations, the functions shown within a block may be executed in an order different from the order shown in each diagram. For example, two consecutively shown blocks may actually be achieved as a single process, executed simultaneously or nearly simultaneously, executed in a partially or entirely overlapping manner in time, or, if applicable, executed in reverse order, depending on the functions involved. Each block in a block diagram or flowchart or both, and combinations of multiple blocks in a block diagram or flowchart or both, may be executed by a dedicated hardware-based system that performs a particular function or operation, or by a combination of dedicated hardware and computer instructions.

[0107] While various embodiments of this disclosure have been described as examples, they are not intended to be exhaustive or limit the scope to these embodiments. As will be apparent to those skilled in the art, many modifications and variations are possible without departing from the scope and spirit of each embodiment described. The terminology used herein has been selected to best describe the principles, practical applications, or technical improvements to the technologies found in the market of each embodiment, or to enable those skilled in the art to understand each embodiment disclosed herein.

Claims

1. A computer implementation method, Acquiring multiple historical data items and multiple new data items by one or more processors, wherein the multiple historical data items are used for training a model and the multiple new data items are applied to the model, The one or more processors determine at least one of the overall difference, structural difference, and confidence difference between the plurality of historical data items and the plurality of new data items, including the structural difference. The one or more processors determine whether to update the model based on at least one of the overall difference, the structural difference, and the reliability difference, including the structural difference. Includes, The one or more processors cluster the multiple historical data items into a first multiple cluster, The above-mentioned one or more processors cluster the above-mentioned multiple new data items into a second plurality of clusters, The one or more processors determine the first number of data items in the plurality of new data items, It further includes, The determination of the aforementioned structural difference is The selection of a second set of new data items by one or more processors, wherein the cluster of each data item in the second set of new data items according to the second set of clusters is the same as the cluster of the first set of clusters, The process involves one or more processors selecting a third set of new data items from a second set of new data items, wherein the structure of each data item in the third set of new data items is different from the structure of the plurality of historical data items. The one or more processors determine the third number of data items in the third set of new data items, The one or more processors determine the structural difference based on the first number and the third number, Computer implementation methods, including those mentioned above.

2. The selection of the third set of the aforementioned new data items is as follows: For each data item in the second set of the aforementioned new data items The one or more processors determine the cluster of the data items, The process involves one or more processors selecting a first set of historical data items from the plurality of historical data items, wherein the first set of historical data items is clustered in the cluster. The one or more processors determine the target value of the data item and the target value distribution of the first set of historical data items. In accordance with the determination that the target value of the data item does not fall within the target value distribution of the first set of historical data items, one or more processors determine that the data item is a data item in the third set of new data items, The computer implementation method according to claim 1, including the method described in claim 1.

3. The determination of the aforementioned target value and the distribution of the aforementioned target value is: The set of weights is determined by one or more of the aforementioned processors, One or more processors weight the data item with the set of weights in order to determine the target value of the data item, One or more processors weight the first set of historical data items by the set of weights in order to determine the target value distribution of the first set of historical data items. The computer implementation method according to claim 2, including the method described in claim 2.

4. A computer implementation method, Acquiring multiple historical data items and multiple new data items by one or more processors, wherein the multiple historical data items are used for training a model and the multiple new data items are applied to the model, The one or more processors determine at least one of the overall difference, structural difference, and confidence difference between the plurality of historical data items and the plurality of new data items, including the confidence difference. The one or more processors determine whether to update the model based on at least one of the overall difference, the structural difference, and the reliability difference, including the reliability difference. Includes, The one or more processors cluster the multiple historical data items into a first multiple cluster, The above-mentioned one or more processors cluster the above-mentioned multiple new data items into a second plurality of clusters, The one or more processors determine the first number of data items in the plurality of new data items, It further includes, The determination of the aforementioned confidence difference is as follows: The selection of a fourth set of new data items by one or more processors, wherein the cluster of each data item in the fourth set of new data items according to the second plurality of clusters is the same as the cluster of the first plurality of clusters. The process involves one or more processors selecting a fifth set of new data items from a fourth set of new data items, wherein the confidence level of each data item in the fifth set of new data items is different from the confidence levels of the multiple historical data items. The one or more processors determine the number of data items in the fifth set of new data items, The reliability difference is determined by one or more processors based on the first number and the fourth number, Computer implementation methods, including those mentioned above.

5. The selection of the fifth set of the aforementioned new data items is as follows: For each data item in the fourth set of the new data items, The one or more processors determine the cluster of the data items, The process involves one or more processors selecting a second set of historical data items from the plurality of historical data items, wherein the second set of historical data items is clustered in the cluster. The one or more processors determine the confidence level of the data item and the confidence interval of the second set of historical data items, In accordance with the determination that the confidence level of the data item does not fall within the confidence interval of the second set of historical data items, one or more processors determine that the data item is a data item in the fifth set of new data items, The computer implementation method according to claim 4, including the method described in claim 4.

6. The determination of the confidence level and the confidence interval is as follows: The one or more processors apply the data items to the model to determine the reliability, The one or more processors apply the second set of historical data items to the model in order to determine the confidence interval, The computer implementation method according to claim 5, including the method described in claim 5.

7. One or more computer-readable storage media comprising program instructions collectively stored in one or more computer-readable storage media, The method includes one or more processors configured to execute program instructions for performing the method, and the method is The acquisition of multiple historical data items and multiple new data items, wherein the multiple historical data items are used for training a model, and the multiple new data items are applied to the model. To determine at least one of the overall difference, structural difference, and confidence difference between the plurality of historical data items and the plurality of new data items, including the structural difference, Based on the overall difference, the structural difference, and the reliability difference, including at least one of the structural differences, a decision is made on whether to update the model. Includes, The above method further, Clustering the aforementioned multiple historical data items into a first multiple cluster, Clustering the aforementioned multiple new data items into a second set of multiple clusters, The first number of data items in the aforementioned plurality of new data items is determined, Includes, The determination of the aforementioned structural difference is The selection involves selecting a second set of new data items from the aforementioned plurality of new data items, wherein the cluster of each data item in the second set of new data items according to the second plurality of clusters is the same as the cluster of the first plurality of clusters. The process involves selecting a third set of new data items from a second set of new data items, wherein the structure of each data item in the third set of new data items is different from the structure of the plurality of historical data items. Determining the third number of data items in the third set of the new data items, The structural difference is determined based on the first number and the third number, A system that includes this.

8. The selection of the third set of the aforementioned new data items is as follows: For each data item in the second set of the aforementioned new data items Determining the cluster of the aforementioned data items, The process involves selecting a first set of historical data items from the aforementioned plurality of historical data items, wherein the first set of historical data items is clustered into the aforementioned cluster. The objective values ​​of the data items and the target value distribution of the first set of historical data items are determined. In accordance with the determination that the target value of the data item does not fall within the target value distribution of the first set of historical data items, it is determined that the data item is a data item in the third set of new data items. The system according to claim 7, including the system described in claim 7.

9. The determination of the aforementioned target value and the distribution of the aforementioned target value is: Determining the set of weights, In order to determine the target value of the aforementioned data item, the data item is weighted by the aforementioned set of weights, In order to determine the target value distribution of the first set of historical data items, the first set of historical data items is weighted by the set of weights, The system according to claim 8, including the above.

10. One or more computer-readable storage media comprising program instructions collectively stored in one or more computer-readable storage media, The method includes one or more processors configured to execute program instructions for performing the method, and the method is The acquisition of multiple historical data items and multiple new data items, wherein the multiple historical data items are used for training a model, and the multiple new data items are applied to the model. To determine at least one of the overall difference, structural difference, and confidence difference between the plurality of historical data items and the plurality of new data items, including the confidence difference; Based on the overall difference, the structural difference, and the reliability difference, including the reliability difference, a decision is made on whether to update the model. Includes, The above method further, Clustering the aforementioned multiple historical data items into a first multiple cluster, Clustering the aforementioned multiple new data items into a second set of multiple clusters, The first number of data items in the aforementioned plurality of new data items is determined, Includes, The determination of the aforementioned confidence difference is as follows: The selection involves selecting a fourth set of new data items from the aforementioned plurality of new data items, wherein the cluster of each data item in the fourth set of new data items according to the second plurality of clusters is the same as the cluster of the first plurality of clusters. The selection involves choosing a fifth set of new data items from a fourth set of new data items, wherein the confidence level of each data item in the fifth set of new data items is different from the confidence levels of the multiple historical data items. To determine the number of data items in the fifth set of the new data items, The confidence difference is determined based on the first number and the fourth number, A system that includes this.

11. The selection of the fifth set of the aforementioned new data items is as follows: For each data item in the fourth set of the new data items, Determining the cluster of the aforementioned data items, The process involves selecting a second set of historical data items from the aforementioned plurality of historical data items, wherein the second set of historical data items is clustered in the aforementioned cluster. The confidence level of the aforementioned data item and the confidence interval of the second set of the aforementioned historical data items are determined. In accordance with the determination that the confidence level of the aforementioned data item does not fall within the confidence interval of the second set of historical data items, it is determined that the aforementioned data item is a data item in the fifth set of new data items, The system according to claim 10, including the following:

12. The determination of the confidence level and the confidence interval is as follows: Applying the data items to the model in order to determine the confidence level, Applying the second set of historical data items to the model in order to determine the confidence interval, The system according to claim 11, including the following:

13. A computer program including program instructions, wherein the program instructions are executable by one or more processors, causing one or more processors to perform an operation, and the operation is The acquisition of multiple historical data items and multiple new data items, wherein the multiple historical data items are used for training a model, and the multiple new data items are applied to the model. To determine at least one of the overall difference, structural difference, and confidence difference between the plurality of historical data items and the plurality of new data items, including the structural difference, Based on the overall difference, the structural difference, and the reliability difference, including at least one of the structural differences, a decision is made on whether to update the model. Includes, Clustering the aforementioned multiple historical data items into a first multiple cluster, Clustering the aforementioned multiple new data items into a second set of multiple clusters, The first number of data items in the aforementioned plurality of new data items is determined, It further includes, The determination of the aforementioned structural difference is The selection of a second set of new data items by one or more processors, wherein the cluster of each data item in the second set of new data items according to the second set of clusters is the same as the cluster of the first set of clusters, The process involves one or more processors selecting a third set of new data items from a second set of new data items, wherein the structure of each data item in the third set of new data items is different from the structure of the plurality of historical data items. The one or more processors determine the third number of data items in the third set of new data items, The one or more processors determine the structural difference based on the first number and the third number, A computer program that includes [this].

14. A computer program including program instructions, wherein the program instructions are executable by one or more processors, causing one or more processors to perform an operation, and the operation is The acquisition of multiple historical data items and multiple new data items, wherein the multiple historical data items are used for training a model, and the multiple new data items are applied to the model. To determine at least one of the overall difference, structural difference, and confidence difference between the plurality of historical data items and the plurality of new data items, including the confidence difference; Based on the overall difference, the structural difference, and the reliability difference, including the reliability difference, a decision is made on whether to update the model. Includes, Clustering the aforementioned multiple historical data items into a first multiple cluster, Clustering the aforementioned multiple new data items into a second set of multiple clusters, The first number of data items in the aforementioned plurality of new data items is determined, It further includes, The determination of the aforementioned confidence difference is as follows: The selection of a fourth set of new data items by one or more processors, wherein the cluster of each data item in the fourth set of new data items according to the second plurality of clusters is the same as the cluster of the first plurality of clusters. The process involves one or more processors selecting a fifth set of new data items from a fourth set of new data items, wherein the confidence level of each data item in the fifth set of new data items is different from the confidence levels of the multiple historical data items. The one or more processors determine the number of data items in the fifth set of new data items, The reliability difference is determined by one or more processors based on the first number and the fourth number, A computer program that includes [this].

Images