Apparatus, method and computer program product for multi-label classification using adjusted multi-class

Abstract

The multi-tag classification system performs multi-tag classification using the adjusted multi-category classification model in combination with pre-classification and post-classification processing and data transformation. An original multi-label training data set in which objects are classified into at least one, but possibly a plurality of original classification sets is transformed into an adjusted multi-class training data set, in which the same objects are each classified into a combined classification set, the combined classification set represents a combination of all individual original classification sets in the original training data set into which the object is classified. A multi-class classification model is trained using the adjusted training dataset, and a multi-class classification of the object is generated for the input dataset using the trained model. The multi-class classification is transformed into a multi-tag classification.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to machine learning classification models and, in particular, in some examples, to machine learning models for multi-label classification of quality management events. Background Art

[0002] Applicants have recognized exemplary technical challenges and difficulties associated with machine learning models for multi-label classification.Through effort, ingenuity, and innovation, applicants have solved problems associated with machine learning models for multi-label classification. Summary of the invention

[0003] According to one aspect, an exemplary embodiment of the present disclosure includes a device, the device including at least one processor and at least one non-volatile memory, the at least one non-volatile memory including program code stored thereon. The at least one non-volatile memory and the program code are configured to use the at least one processor to enable the device to at least: transform an original training data set corresponding to a multi-label classification task into an adjusted training data set corresponding to the multi-class classification task based at least in part on a predefined transformation protocol, wherein a multi-class classification model configured to perform the multi-class classification task is trained at least in part based on the adjusted training data set; transform multi-class output data generated for an input data set using a trained multi-class classification model into multi-label output data corresponding to the multi-label classification task based at least in part on the predefined transformation protocol; and perform at least one enterprise management operation based at least in part on the multi-label output data.

[0004] In some embodiments, the original training data set includes a multi-label classification of training objects, each training object is classified into at least one original classification set in a plurality of original classification sets, wherein at least one of the training objects is classified into a plurality of different original classification sets. The adjusted training data set may include a multi-category classification of the training objects, each training object is classified into a corresponding combination classification set in at least one combination classification set, wherein the combination classification set corresponding to each specific training object among the training objects represents a combination of each original classification set to which the specific training object is classified in the multi-label classification of the original training data set. The multi-category output data may include a classification of production objects from the input data set, each production object is classified into a corresponding combination classification set in the at least one combination classification set from the adjusted training data set. The multi-label output data may include a classification of the production objects from the input data set, each production object is classified into at least one original classification set from the plurality of original classification sets from the original training data set.

[0005] In some embodiments, the at least one non-volatile memory and the program code are configured to, with the at least one processor, further cause the apparatus to at least: train the multi-class classification model based at least in part on the adjusted training data set.

[0006] In some embodiments, the at least one non-transitory memory and the program code are configured to, with the at least one processor, further cause the apparatus to at least: generate the multi-category output data for the input data set using a trained multi-category classification model.

[0007] In some embodiments, transforming the original training data set into the adjusted training data set includes: for each specific training object among the training objects classified in the original training data set, combining all original classification labels corresponding to the original classification set into which the specific training object in the original training data set is classified into a combined classification label, wherein the combined classification label includes all of these original classification labels for the specific training object delimited by the label combiner operator defined according to the predefined transformation protocol.

[0008] In some embodiments, transforming the multi-category output data into the multi-label output data includes: for each specific production object among the production objects classified in the multi-category output data, at least partially based on a label combiner operator defined according to the predefined transformation protocol, parsing a combined classification label corresponding to a combined classification set into which the specific production object in the multi-category output data is classified into at least one original classification label corresponding to an original classification set from the original training data set into which the specific production object in the multi-label output data will be classified, wherein the combined classification label includes the at least one original classification label delimited by the label combiner operator.

[0009] In some embodiments, the multi-label classification task includes classifying a text object containing text characterizing a quality management event associated with a product produced by an enterprise into a classification set corresponding to medical terms from a medical dictionary.

[0010] According to another aspect, an embodiment of the present invention is characterized in a method, comprising: transforming an original training data set corresponding to a multi-label classification task into an adjusted training data set corresponding to the multi-category classification task at least partially based on a predefined transformation protocol, wherein a multi-category classification model configured to perform the multi-category classification task is trained at least partially based on the adjusted training data set; transforming multi-category output data generated for an input data set using the trained multi-category classification model into multi-label output data corresponding to the multi-label classification task at least partially based on the predefined transformation protocol; and performing at least one enterprise management operation based at least partially on the multi-label output data.

[0011] According to another aspect, an embodiment of the present invention is characterized by a computer program product including at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein. The computer-readable program code portions include an executable portion configured to: transform an original training data set corresponding to a multi-label classification task into an adjusted training data set corresponding to the multi-class classification task based at least in part on a predefined transformation protocol, wherein a multi-class classification model configured to perform the multi-class classification task is trained based at least in part on the adjusted training data set; transform multi-class output data generated for an input data set using the trained multi-class classification model into multi-label output data corresponding to the multi-label classification task based at least in part on the predefined transformation protocol; and cause at least one enterprise management operation to be performed based at least in part on the multi-label output data.

[0012] The above-mentioned summary of the invention is provided only for the purpose of summarizing some exemplary embodiments, so as to provide a basic understanding of some aspects of the present disclosure. Therefore, it should be understood that the above-mentioned embodiments are only examples and should not be construed as narrowing the scope or essence of the present disclosure in any way. It should be understood that, in addition to those summarized here, the scope of the present disclosure also covers many possible embodiments, some of which will be further described below. Other features, aspects and advantages of this theme will be apparent from the description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Having thus generally described embodiments of the present disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which:

[0014] Figure 1 An exemplary block diagram illustrating an environment in which embodiments of the present disclosure may operate;

[0015] Figure 2 An exemplary block diagram showing an exemplary apparatus that may be specially configured according to an exemplary embodiment of the present disclosure;

[0016] Figure 3 An exemplary multi-label classification system according to at least some exemplary embodiments of the present disclosure is shown;

[0017] Figure 4 shows exemplary data objects used and generated by a multi-label classification system at various stages according to at least some exemplary embodiments of the present disclosure; and

[0018] Figure 5is a flow chart depicting an example process for performing multi-label classification using a tuned multi-class classification model according to at least some example embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Some embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some but not all embodiments of the present disclosure are shown. Indeed, the embodiments of the present disclosure can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will satisfy applicable legal requirements. Throughout the text, like reference numerals refer to like elements.

[0020] As used herein, terms such as "front," "rear," "top," and the like are used for illustrative purposes in the examples provided below to describe the relative positions of certain components or portions of components. Additionally, as will be apparent to one of ordinary skill in the art from this disclosure, the terms "substantially" and "approximately" indicate that the referenced element or associated description is accurate within an applicable engineering tolerance.

[0021] As used herein, the term "comprising" means including but not limited to, and should be interpreted in the manner commonly used in the patent context. It should be understood that the use of broad terms such as "comprising," "including," and "having" provides support for narrower terms such as "consisting of," "consisting essentially of," and "consisting essentially of."

[0022] The phrases "in one embodiment," "according to one embodiment," "in some embodiments," and the like generally mean that the particular feature, structure, or characteristic following the phrase may be included in at least one embodiment of the present disclosure, and may be included in more than one embodiment of the present disclosure (importantly, such phrases are not necessarily referring to the same embodiment).

[0023] The word “example” or “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

[0024] If the specification states that a component or feature "may", "can", "could", "should", "will", "preferably", "likely", "usually", "optionally", "for example", "often", or "might" (or other such language) be included or have a property, the specific component or feature is not required to be included or have that property. Such components or features may optionally be included in some embodiments, or may be excluded.

[0025] The use of the term "circuit" as used herein with respect to the components of a system or device should be understood to include specific hardware configured to perform functions associated with a specific circuit as described herein. The term "circuit" should be broadly understood to include hardware, and in some embodiments, software for configuring hardware. For example, in some embodiments, a "circuit" may include a processing circuit, a communication circuit, an input / output circuit, etc. In some embodiments, other elements may provide or supplement the functionality of a specific circuit. Alternatively or in addition, in some embodiments, other elements of the system and / or device described herein may provide or supplement the function of another specific circuit group. For example, a processor may provide processing functionality to any circuit group in a circuit group, a memory may provide storage functionality to any circuit group in a circuit group, and a communication circuit may provide network interface functionality to any circuit group in a circuit group, etc.

[0026] The terms “electrically coupled,” “electrically coupled,” “in communication with,” “in electronic communication with,” or “connected” in this disclosure refer to two or more elements or components that are connected by wired means and / or wireless means such that signals, voltage / current, data and / or information can be transmitted to and / or received from these elements or components.

[0027] Machine learning systems have been proposed to solve classification problems within a variety of contexts and with respect to a variety of subject areas. In these classification problems, in some examples, objects represented in data are classified into categories based on features of the data associated with the objects being classified. For purposes of illustration, in some examples, text objects containing various discrete text descriptions may be classified into categories based on features of the text objects (such as, for example, the presence of certain words in the text, patterns of how certain words or types of words are arranged relative to each other, semantic and / or numerical properties derived from the text, and / or various data and / or metadata attributes associated with the text objects). A (e.g., annotated) training dataset including sample text objects each associated with a classification label may be used (e.g., in conjunction with a machine learning algorithm) to develop, define, and / or train a classification model that is capable of classifying unlabeled and / or unclassified text objects of a given input dataset into various categories (or classification sets) reflected in the training dataset.

[0028] The types of classification problems that these machine learning systems may target include multi-label classification and multi-class classification, to name a few examples. In multi-label classification, a given object can be classified into multiple, non-exclusive classification sets, e.g., multiple possible classification labels representing different classification sets are assigned to the object. On the other hand, in multi-class classification, a given object can be classified into exactly one classification set out of at least three classification sets.

[0029] In some exemplary domain-specific scenarios, enterprises such as commerce or other organizations can produce various types of products (e.g., manufactured goods, provided services) via various research, development, manufacturing, distribution and / or service provision processes associated with the production of these products. In many such cases, these enterprises can develop, implement and / or execute quality management systems, processes and / or operations about these research, development, manufacturing, distribution and / or service provision processes, with the aim of ensuring that the quality of the products being produced meets specific standards at a specific consistency level. Sometimes, some aspects of this quality management may even be required by laws and / or regulations, and some quality management systems, processes and / or operations can be designed and / or configured to ensure that the research, development, manufacturing, distribution and / or service provision processes associated with the production of these products comply with applicable laws and / or regulations. In particular, quality management may be particularly important for the production of certain types of products (including medical devices, drugs and / or medical or life science services). For example, these products may be subject to relatively high levels of quality management regulations.

[0030] Quality management software system (also referred to as quality management system in this article) can be used to implement and / or promote the quality management measures taken by enterprises for the production of their products and / or the provision of their services. More specifically, in one example, the quality management system is likely to promote the process of receiving, generating and / or gathering any data associated with the production of products required for quality management purposes. The quality management system can be configured to record quality management events based on inputs from various sources, and the input may include internal enterprise data associated with the production of related products, data generated and / or received in conjunction with the supply or use of related products, and / or data characterizing the communication of various entities associated with the production, consumption and / or use of related products and / or data derived from the communication. In one example, the system can be configured to enable users to generate and fill in various types of forms associated with quality management, including complaint forms, supplier quality management forms, audit forms, risk forms and / or quality event forms, just to name a few examples. The data can be collected, archived and consulted to provide visibility of various quality management goals and / or operations related to various research, development, manufacturing, distribution and / or service provision processes subject to quality management constraints, and / or initiate and / or enable the execution of various quality management operations on the process in question. In some examples, the quality management system can be configured to present status data and / or generate and present insights (e.g., predictions and / or recommendations) for optimizing the development and execution of quality management operations. These insights are typically generated using machine learning models, which can be developed, configured, and / or trained using one or more machine learning algorithms.

[0031] Some data that may be collected by a quality management system (e.g., quality management events) may include text, including, for example, text that characterizes how a product is provided, consumed, and / or used, text that characterizes the specific context in which a product is provided, consumed, and / or used, and / or text that characterizes certain effects resulting from the provision, consumption, and / or use of a product, among other examples. The text objects (e.g., included by a quality management event) may contain text that represents and / or describes customer complaints arising from the provision, consumption, and / or use of a product that is subject to quality management constraints of the quality management system. In certain domains or contexts, it may be valuable or useful to perform classification tasks on these text objects. In one example, it may be beneficial for a quality management system to classify each of these text objects into one or more classification sets that represent terms defined in the Medical Dictionary for Regulatory Activities (MedDRA), which is a dictionary of standard terms that define various symptoms, signs, diseases, syndromes, and diagnoses, to name just a few examples. In this case, the quality management system is likely configured to perform multi-label classification (e.g., using a machine learning model and / or algorithm) for the text objects and MedDRA terms, assigning at least one (but potentially multiple) classification labels to each text object based on features of the text object, each classification label corresponding to one of the MedDRA terms. For purposes of illustration, in one example, text from a quality management event describing a cough may be assigned classification labels representing multiple MedDRA terms associated with a cough, such as "flu," "cancer," and any other terms that may be associated with a cough. Thus, for example, quality management data may be enhanced as a result of this multi-label classification because any MedDRA terms that may be associated with a particular text object may be automatically determined and associated with the text (or a quality management event containing the text), and these associations may facilitate further analysis and / or interpretation of the data and / or performance of quality management operations associated with the data.

[0032] However, there are many challenges with existing multi-label classification methods. For example, in some cases, it has been proposed to implement multi-label classification using multiple specialized binary classification models, each of which is independently trained with respect to a separate classification label, thereby requiring each of these binary classification models (via a corresponding specialized algorithm) to be used for each given object to be classified. The scalability of this approach is limited, and performing multi-label classification with respect to a sufficiently large number of classification labels may be time-consuming or may consume excessive processing and memory resources. These shortcomings may make multi-label classification impractical or even impossible in some cases.

[0033] In various embodiments, the currently disclosed multi-label classification system is configured to perform a multi-label classification task using an adjusted multi-class classification model in combination with pre-classification and post-classification processing and data transformation steps. More specifically, in one example, the original training data set corresponding to the multi-label classification task is transformed into an adjusted training data set, wherein any individual classification labels assigned to a specific object are combined into a single combined label, which includes all individual classification labels delimited via a combiner operator. The adjusted training data set is then used to train the multi-class classification model, and the trained model is used to generate a multi-class prediction about the combined label reflected in the adjusted training data. The combined labels indicated in these multi-class predictions are then parsed and transformed into multi-label predictions (e.g., based on a combiner operator, using a mapping of the combined label to its corresponding individual label). Therefore, the multi-label classification system only involves a single (multi-class) classification task instead of multiple binary classification tasks, thereby improving performance, reducing implementation complexity and resource consumption, expanding the scope of application of related multi-label classification functions, and eliminating the need for multiple dedicated algorithms and binary classification models and the need for non-standard accuracy and cost metrics for measuring performance and for training models, as well as other examples.

[0034] Figure 1 An exemplary block diagram of an environment 100 in which embodiments of the present disclosure may operate is shown. Specifically, Figure 1 Shown are a multi-label classification system 140 , a quality management system 120 , one or more data repositories 150 , and one or more user devices 160 , all connected to a network 130 .

[0035] The network 130 can be embodied in any of numerous network configurations. In some embodiments, the network 130 can be a public network (e.g., the Internet). In some embodiments, the network 130 can be a private network (e.g., an internal localized closed network between specific devices). In some other embodiments, the network 130 can be a hybrid network (e.g., a network that enables internal communication between specific connected devices and external communication with other devices). In various embodiments, the network 130 can include one or more base stations, repeaters, routers, switches, signal towers, communication cables, routing stations, etc. In various embodiments, the components of the environment 100 can be communicatively coupled to transmit data to each other and / or receive data from each other through the network 130. Such configurations include, but are not limited to, wired or wireless personal area networks (PANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), etc.

[0036] One or more data repositories 150 may be configured to receive, store and / or transmit data. In some embodiments, one or more data repositories 150 may be configured to store enterprise data associated with one or more enterprises producing one or more products (e.g., quality management data, product data, process data, system data, any data that can be used in conjunction with quality management operations, including any data that can be included in and / or used to generate a training data set for a multi-label classification model, and / or any related object data and / or metadata). In some embodiments, one or more data repositories 150 may be configured to store data associated with the development, configuration, training and / or use (e.g., by a multi-label classification system 140) of one or more machine learning models corresponding to multi-label and / or multi-class prediction tasks associated with quality management of product production, including, for example, original and / or adjusted training data, transformation protocols for transforming training data and / or output data, defining trained model weights, parameters and / or hyperparameters for trained classification models, and multi-class and / or multi-label output data generated via trained classification models. In various embodiments, some or all of the data stored in one or more data repositories 150 may be stored in a shared memory system shared between the quality management system 120 and / or the multi-label classification system 140, and / or may be otherwise accessible to the quality management system 120 and the multi-label classification system 140.

[0037] One or more user devices 160 may be associated with and / or operated by users of the quality management system 120, the multi-label classification system 140, and / or one or more enterprise systems associated with one or more products and / or the production of one or more products. In one example, an enterprise system associated with one or more products and / or the production of products may cause data associated with the production of one or more products to be transmitted to, displayed on, and / or used to generate data or interface elements displayed on the user device 160, and / or may cause one or more interfaces (e.g., user interfaces) to be presented via the user device 160, which may be configured to receive input including and / or used to generate any data associated with one or more products and / or the production of products, including, in some examples, product data, process data, and / or system data. In another example, the quality management system 120 may cause data associated with the development, configuration, monitoring, tracking, optimization, and / or execution of quality management operations and / or objectives for the production of a product to be transmitted to, displayed on, and / or used to generate data or interface elements displayed on the user device 160, and / or may cause one or more interfaces (e.g., user interfaces) to be presented via the user device 160, which may be configured to receive input including and / or used to generate any data associated with the quality management operations and / or objectives. In another example, the multi-label classification system 140 may cause data associated with the development, configuration, training, and / or use of one or more machine learning models corresponding to multi-label and / or multi-class prediction tasks associated with the quality management of the production of a product to be transmitted to, displayed on, and / or used to generate data or interface elements displayed on the user device 160, and / or may cause one or more interfaces (e.g., user interfaces) to be presented via the user device 160, which may be configured to receive input including and / or used to generate any data associated with the machine learning model.

[0038] The multi-label classification system 140 can be a computing system or device (e.g., a server system) configured via hardware, software, firmware, and / or a combination thereof to perform multi-label classification functionality, including, for example, transforming original training data into adjusted training data, training a classification model, generating output data using the trained model, transforming multi-category output data into multi-label output data, and / or performing enterprise management operations (e.g., quality management operations) based on the output data.

[0039] The quality management system 120 can be a computing system or device (e.g., a server system) configured via hardware, software, firmware, and / or a combination thereof to perform quality management operations regarding the production of a product, including, for example, presenting an interface configured to receive inputs used in the performance of the quality management operations; receiving (e.g., via a presented interface) inputs used in the performance of the quality management operations; generating, receiving, accessing, retrieving, and / or aggregating quality management events and / or any event data included therein or derived therefrom; and / or performing quality management operations based on retrieved event data and / or quality management events. In some embodiments, the quality management system 120 can correspond to and / or include a quality management software system and / or a quality management system that is similar to and / or performs similar functions to the quality management software system and / or quality management system defined and described above.

[0040] In various embodiments, the quality management system 120 and the multi-label classification system 140 may have different degrees of integration with respect to each other. For example, the quality management system 120 and the multi-label classification system 140 may be configured to transmit and / or receive communications and / or data with respect to each other, and / or access a commonly accessible or shared data repository to exchange training data, input data, and / or output data associated with a classification model. In another example, the multi-label classification system 140 may be a subsystem of the quality management system 120 and / or otherwise integrated as part of the quality management system. In another example, the quality management system 120 and the multi-label classification system 140 may be a subsystem of a larger enterprise management system or enterprise performance management system configured to perform one or more enterprise management operations (e.g., including quality management operations) for one or more products produced by one or more enterprises and / or one or more processes and / or systems associated with the production of the one or more products and / or may be otherwise integrated as part of the larger enterprise management system or enterprise performance management system.

[0041] Although Figure 1 Certain components are shown as separate, independent entities communicating via network 130, but various embodiments are not limited to this configuration. In other embodiments, one or more components may be directly connected and / or shared hardware, etc.

[0042] Figure 2 An exemplary block diagram of an exemplary device that can be specially configured according to an exemplary embodiment of the present disclosure is shown. Specifically, Figure 2An exemplary computing device 200 ("device 200") that is specially configured in accordance with at least some exemplary embodiments of the present disclosure is depicted. Examples of device 200 may include, but are not limited to, one or more components of one or more operating systems associated with one or more products and / or the production of one or more products, a multi-label classification system 140, a quality management system 120, a data repository 150, and / or a user device 160. Device 200 includes a processor 202, a memory 204, an input / output circuit 206, a communication circuit 208, and / or a transformation circuit 210, a training circuit 212, a classification circuit 214, and / or an integrated circuit 216. In some embodiments, device 200 is configured to perform and carry out the operations described herein.

[0043] Although these components are described with respect to functional limitations, it should be understood that a particular implementation necessarily includes the use of specific computing hardware. It should also be understood that in some embodiments, certain components described herein include similar or common hardware. For example, in some embodiments, both sets of circuits utilize the use of the same processor, memory, circuits, etc. to perform their associated functions, such that each set of circuits does not require duplicate hardware.

[0044] In various embodiments, a device, system or apparatus (such as an apparatus 200 of one or more components of one or more operating systems associated with one or more products and / or the production of one or more products, a multi-label classification system 140, a quality management system 120, a data repository 150, and / or a user device 160) may refer to, for example, one or more computers, computing entities, desktop computers, mobile phones, tablet computers, small tablets, notebooks, laptop computers, distributed systems, servers, etc., and / or any combination of devices or entities suitable for performing the functions, operations and / or processes described herein. Such functions, operations and / or processes may include, for example, transmission, reception, operation, processing, display, storage, determination, creation / generation, monitoring, evaluation, comparison, and / or similar terms that may be used herein. In one embodiment, these functions, operations and / or processes may be performed on data, content, information, and / or similar terms that may be used herein. In this regard, as described herein, the apparatus 200 embodies a specifically configured specific computing entity that is transformed to enable specific operations described herein and provide specific advantages associated therewith.

[0045] The processor 202 or processor circuit 202 can be embodied in a variety of different ways. In various embodiments, the use of the term "processor" should be understood to include a single-core processor, a multi-core processor, multiple processors within the device 200, and / or one or more remote or "cloud" processors outside the device 200. In some exemplary embodiments, the processor 202 may include one or more processing devices configured to execute independently. Alternatively or in addition, the processor 202 may include one or more processors configured in series via a bus to achieve independent execution of operations, instructions, pipelines, and / or multithreading.

[0046] In an exemplary embodiment, the processor 202 may be configured to execute instructions stored in the memory 204 or otherwise accessible by the processor. Alternatively or in addition, the processor 202 may be configured to perform hard-coded functions. Therefore, whether configured by hardware methods or software methods, or by a combination thereof, the processor 202 may represent an entity (e.g., physically embodied in circuit form) that is capable of performing operations according to the embodiments of the present disclosure while being configured accordingly. Alternatively or in addition, the processor 202 may be embodied as an executor of software instructions, and these instructions may specifically configure the processor 202 to perform various algorithms embodied in one or more operations described herein when executing such instructions. In some embodiments, the processor 202 includes hardware, software, firmware, and / or a combination thereof that performs one or more operations described herein.

[0047] In some embodiments, the processor 202 (and / or a coprocessor or any other processing circuitry assisting the processor or otherwise associated with the processor) communicates with the memory 204 via a bus for transferring information between components of the device 200.

[0048] The memory 204 or the memory circuit embodying the memory 204 may be non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In some embodiments, the memory 204 includes or embodies an electronic storage device (e.g., a computer-readable storage medium). In some embodiments, the memory 204 is configured to store information, data, content, applications, instructions, etc. for enabling the apparatus 200 to perform various operations and / or functions according to the exemplary embodiments of the present disclosure.

[0049] Input / output circuit 206 may be included in device 200. In some embodiments, input / output circuit 206 may provide output to a user and / or receive input from a user. Input / output circuit 206 may communicate with processor 202 to provide such functionality. Input / output circuit 206 may include one or more user interfaces. In some embodiments, the user interface may include a display including an interface presented as a web user interface, an application user interface, a user device, a back-end system. In some embodiments, input / output circuit 206 also includes a keyboard, a mouse, a joystick, a touch screen, a touch area, a soft key, a microphone, a speaker or other input / output mechanism. Processor 202 and / or input / output circuit 206 including a processor may be configured to control one or more operations and / or functions of one or more user interface elements by computer program instructions (e.g., software and / or firmware) stored on a memory accessible to the processor (e.g., memory 204, etc.). In some embodiments, input / output circuit 206 includes or utilizes a user-oriented application to provide input / output functionality to a computing device and / or other display associated with a user.

[0050] The communication circuit 208 may be included in the device 200. The communication circuit 208 may include any device, such as a device or circuit embodied in hardware or a combination of hardware and software, which is configured to receive and / or transmit data from / to a network and / or any other device, circuit or module that communicates with the device 200. In some embodiments, the communication circuit 208 includes, for example, a network interface for implementing communication with a wired or wireless communication network. In addition or alternatively, the communication circuit 208 may include one or more network interface cards, antennas, buses, switches, routers, modems and supporting hardware, firmware and / or software, or any other device suitable for implementing communication via one or more communication networks. In some embodiments, the communication circuit 208 may include a circuit for interacting with an antenna and / or other hardware or software to cause a signal to be transmitted via an antenna and / or to process a signal received via an antenna. In some embodiments, the communication circuit 208 enables data to be transmitted and / or received to and / or from a user device and / or other external computing device that communicates with the device 200.

[0051] The transformation circuit 210 may be included in the apparatus 200 (e.g., an apparatus specifically corresponding to the multi-label classification system 140). The transformation circuit 210 may include hardware, software, firmware, and / or a combination thereof, designed and / or configured to request, receive, process, generate, and transmit data, data structures, control signals, and electronic information for adjusting the multi-class classification model to perform a multi-label classification task, including performing any of the operations described herein with respect to generating adjusted training data and / or generating multi-label output data.

[0052] The training circuit 212 may be included in the apparatus 200 (e.g., an apparatus specifically corresponding to the multi-label classification system 140). The training circuit 212 may include hardware, software, firmware, and / or a combination thereof, designed and / or configured to request, receive, process, generate, and transmit data, data structures, control signals, and electronic information for functionality related to training a classification model, including performing any of the operations described herein with respect to training a multi-class classification model (e.g., using an adjusted training data set).

[0053] The classification circuit 214 may be included in the apparatus 200 (e.g., an apparatus specifically corresponding to the multi-label classification system 140). The classification circuit 214 may include hardware, software, firmware, and / or a combination thereof, designed and / or configured to request, receive, process, generate, and transmit data, data structures, control signals, and electronic information for functionality associated with performing classification tasks, including performing any of the operations described herein with respect to generating multi-class output data using a trained multi-class classification model.

[0054] The integrated circuit 216 may be included in the apparatus 200 (e.g., an apparatus specifically corresponding to the multi-label classification system 140 and / or the quality management system 120). The integrated circuit 216 may include hardware, software, firmware, and / or a combination thereof, designed and / or configured to request, receive, process, generate, and transmit data, data structures, control signals, and electronic information for functionality related to integration between the multi-label classification system 140 and the quality management system 120, including performing any of the operations described herein with respect to communication between the systems: accessing a common data repository, generating data by one system for interpretation, parsing, and / or use by another system, and / or causing the quality management system 120 to perform quality management operations.

[0055] In some embodiments, two or more of the circuits 202-216 groups are combinable. Alternatively or in addition, one or more of the circuits 202-216 groups perform some or all of the operations and / or functionality described herein as being associated with another circuit. In some embodiments, two or more of the circuits 202-216 groups are combined into a single module embodied in the form of hardware, software, firmware, and / or a combination thereof. For example, in some embodiments, one or more of the circuit groups (e.g., conversion circuit 210) can be combined with processor 202 so that processor 202 performs one or more of the operations described herein with respect to conversion circuit 210.

[0056] Figure 3 1 shows a diagram of an exemplary multi-label classification system 140 according to at least some exemplary embodiments of the present disclosure. Specifically, Figure 3 The present invention includes a schematic depiction of an exemplary multi-label classification system 140, including exemplary internal processes and components of the multi-label classification system 140, and / or a schematic depiction of one or more data repositories 150 and a quality management system 120 in communication with the multi-label classification system 140, as well as exemplary data objects used and / or generated by the depicted processes, components, and / or systems. The multi-label classification system 140 and / or an apparatus 200 associated therewith may be specifically configured, for example, via hardware, software, firmware, and / or a combination thereof, to perform operations related to Figure 3 Various data processing and interactions are described to perform multi-label classification using an adapted multi-class classification model and an adapted training dataset with respect to one or more products produced by one or more businesses.

[0057] In the illustrated example, in some examples, the multi-label classification system 140 includes a pre-classification transformation process 312, a post-classification transformation process 332, a training process 322 corresponding to the multi-class classification model 320, and a prediction process 324 corresponding to the multi-class classification model 320. Additionally, in some examples, the one or more data repositories 150 include one or more enterprise data repositories 304 and / or an original training data set 310.

[0058] In an example scenario, enterprise data for one or more enterprises may be generated, collected, captured, received, accumulated, and / or stored in conjunction with one or more products produced by the enterprise, one or more processes associated with the production of the products, and / or one or more operating systems associated with the processes and / or products. Such enterprise data may be stored in one or more enterprise data repositories 304 associated with the enterprise.

[0059] In various embodiments, the enterprise data may include any data associated with one or more products produced by one or more enterprises, including product data, process data, system data, operational data, any data that may be used in conjunction with quality management operations (including quality management data), and / or any data that may be included in a training dataset for a multi-label classification model and / or used to generate the training dataset, and / or any related object data and / or metadata, to name a few examples.

[0060] In one example, product data for enterprise data stored in enterprise data repository 304 may include various data items and / or metadata that identify and / or characterize products produced by the enterprise associated with the enterprise data, including measured, calculated, sensed, and / or otherwise determined values ​​corresponding to various properties associated with the products in general and / or specific instances and / or batches of products that have been produced, to name just a few examples.

[0061] In another example, process data of enterprise data stored in an enterprise data repository may include various data items and / or metadata that identify, define and / or characterize processes associated with the production of a product (e.g., a research process, a manufacturing process, a testing process, a distribution process, a quality management process, a business process), including measured, calculated, sensed, manually configured and / or otherwise determined values ​​corresponding to various attributes associated with the process, to name a few examples.

[0062] In another example, system data of enterprise data stored in an enterprise data repository may include various data items and / or metadata that identify, define, establish configuration parameters for, and / or characterize an operating system associated with a process and / or product (e.g., any computing, mechanical, automation, monitoring, control, laboratory, manufacturing, and / or distribution system or equipment), including measured, calculated, sensed, manually configured, and / or otherwise determined values ​​corresponding to various properties associated with the operating system, the environment in which the system is installed, components included in the system (e.g., devices, assets), configuration parameters that define the functionality of the system and / or its components, and / or operating sessions during previous operation of the system, to name a few examples.

[0063] In another example, the object data and / or metadata of the enterprise data stored in the enterprise data repository 304 may include various data items and / or metadata that identify, define and / or characterize various real or virtual objects associated with a product or the production of a product and / or any relationships between such objects.

[0064] In another example, in some embodiments, the aggregated quality management data 302 of enterprise data stored in the enterprise data repository 304 may include data received, generated, and / or aggregated by and / or within the quality management system 120, including any data associated with the production of a product required or utilized for quality management purposes. In some embodiments, the aggregated quality management data 302 may include quality management events (e.g., generated and / or recorded by the quality management system based on inputs from various sources, which may include internal enterprise data associated with the production of related products, data generated and / or received in conjunction with the supply or use of related products, and / or data representing communications with and / or derived from various entities associated with the production, consumption, and / or use of related products). In one example, quality management events of the aggregated quality management data 302 may include a text object having text characterizing how a product is provided, consumed, and / or used, text characterizing the specific context in which the product is provided, consumed, and / or used, text characterizing certain effects resulting from the provision, consumption, and / or use of the product, as well as other examples, and / or text representing, characterizing, and / or describing customer complaints resulting from the provision, consumption, and / or use of a product subject to quality management constraints of the quality management system 120.

[0065] Although the examples shown involve text objects, and in particular text objects containing text from quality management events from aggregated quality management data 302 aggregated by and / or within the quality management system 120, it should be understood that the present disclosure is applicable to any type of data on which multi-label classification is possible.

[0066] In some embodiments, the enterprise data stored in the enterprise data repository 304 (including the aggregated quality management data 302) may include input related to the production of products and / or processes involving the production of products received from one or more users (e.g., via a user interface presented on a user device 160) and / or may be generated based at least in part on output. In one example, the aggregated quality management data 302 may include communications characterizing and / or data derived from various entities (e.g., customers) associated with the production, consumption, and / or use of related products, including data generated based on input (e.g., user input) received via various types of forms associated with quality management operations and / or presented via the quality management system 120, including complaint forms, supplier quality management forms, audit forms, risk forms, and / or quality incident forms, to name a few examples.

[0067] In addition, in various embodiments, the enterprise data stored in the enterprise data repository 304 may be associated with any stage and / or any corresponding process of the production of a product (including any research, manufacturing, testing, distribution, quality management, and / or commercial stage or process), and / or generated during or in conjunction with any stage and / or any corresponding process of the production of a product.

[0068] In various embodiments, any enterprise data stored in the enterprise data repository 304 (including the aggregated quality management data 302) can be included in the original training data set 310 corresponding to the multi-label classification task and / or used to derive the original training data set. The original training data set 310 can include a multi-label classification of training objects (e.g., corresponding to enterprise data stored in the enterprise data repository and / or data objects derived from the enterprise data), each training object being classified into at least one of a plurality of original classification sets. In one example, at least one of the training objects included in the original training data set 310 can be classified into a plurality of different original classification sets. In some embodiments, the multi-label classification of the training objects in the original data set 310 can be provided via the training annotation data 308 included in the original training data set 310 associated with the training objects. For example, each specific training object included in the original training data set 310 can be associated with at least one classification label (e.g., provided in the training annotation data 308), each classification label corresponding to and / or representing the original classification set to which the specific training object is classified.

[0069] In some embodiments, the original training data set 310 may include and / or may be derived from sample event data 306 from the enterprise data repository 304. The sample event data 306 may include quality management events (e.g., from the aggregated quality management data 302), including, for example, text objects containing text descriptions or narratives that, in some examples, characterize how a product is provided, consumed, and / or used, characterize a specific context in which a product is provided, consumed, and / or used, characterize certain effects resulting from the provision, consumption, and / or use of a product, and other examples, and / or represent, characterize, and / or describe customer complaints resulting from the provision, consumption, and / or use of a product subject to quality management by the quality management system 120. The training objects of the original training data set 310 may correspond to and / or include text objects from the sample event data 306. Furthermore, in some embodiments, each of the original classification labels (of the training annotation data 308) may correspond to and / or represent a particular MedDRA term (as defined and described above), such that each of the training objects (e.g., text objects) may be associated with at least one MedDRA term in the original training data set 310. For example, an association between a text object (of the sample event data 306) and a MedDRA term (of the training annotation data 308) may indicate a determined relevance of the associated MedDRA term to the content of the text object (e.g., the condition and / or situation described within the text of the text object). In some embodiments, the training annotation data 308 may be generated based at least in part on user input (e.g., received via a user interface) regarding the text object in the sample event data 306, in one example, such user input is received from a user who is a subject matter expert. In some embodiments, the training annotation data 308 may be automatically generated (e.g., via one or more analysis algorithms) and / or may be inherent to the sample event data 306 (e.g., corresponding to features or attributes that already exist in the sample event data 306). Additionally or alternatively, the original training dataset 310 may include data sufficient to train a multi-label classification model to perform a multi-label classification task.

[0070] In various embodiments, the multi-label classification system 140 can be configured to perform a multi-label classification task on a given input data set by using the adjusted multi-class classification model 320 .

[0071] More specifically, the multi-label classification system 140 can be configured to perform a multi-label classification task by transforming the multi-label training data into multi-class training data, using the adjusted multi-class classification model 320 in combination with pre-classification processing functionality for adjusting the multi-label training data (e.g., the original training data set 310) for use in the multi-class classification model 320. For example, in some embodiments, the pre-classification transformation process 312 of the multi-label classification system 140 can be configured to receive, retrieve and / or access the original training data 314 (from the original training data set 310 stored in one or more data repositories 150). The original training data 314 can correspond to the data included by the original training data set 310, and / or can include some or all of the data of the original training data set 310 as defined and described above. The pre-classification transformation process 312 can be configured to transform the original training data 314 into the adjusted training data 318, for example, based at least in part on a predefined transformation protocol 316.

[0072] In some embodiments, the transformation protocol 316 may include data defining specific values ​​and / or instructions that determine specific transformation behaviors and / or functionality of the pre-classification transformation process 312 and the post-classification transformation process 332. In one example, the transformation protocol 316 may define a specific format of the adjusted training data 318 expected by the multi-class classification model 320 and / or a specific format of the multi-label output data 334 ultimately produced by the multi-label classification system 140.

[0073] In some embodiments, the adjusted training data 318 may include a multi-class classification for each of the training objects in the original training data 314, with each training object being classified into a corresponding combined classification set (e.g., a set from at least one combined classification set). In one example, the combined classification set corresponding to each particular training object in the adjusted training data 318 may represent a combination of each of the original classification sets into which the particular training object was classified in the multi-label classification provided in the original training data 314. In other words, a training object that was classified into multiple individual classification sets in the original training data 314 may be transformed so that in the adjusted training data 318, the training object is classified into a single combined classification set that collectively represents all of the multiple individual classification sets.

[0074] In one example, transforming the original training data 314 into the adjusted training data 318 (e.g., by the pre-classification transformation process 312) may include: for each specific training object in the training objects classified in the original training data 314, combining all original classification labels corresponding to the original classification set to which the specific training object in the original training data 314 is classified into a combined classification label, the combined classification label including all original classification labels for the specific training object. In some embodiments, the transformation protocol 316 may define a label combiner operator (e.g., by specifying a specific character as a label combiner operator), which label combiner operator may represent a delimiter between separate original classification labels in the combined classification label. The pre-classification transformation process 312 may be configured to generate a combined classification label for a specific training object to include each separate original classification label associated with the specific training object, wherein an instance of the label combiner operator is inserted between each pair of separate original classification labels. In various embodiments, the pre-classification transformation process 312 can be configured to produce an interpretable combined classification label (e.g., by the post-classification transformation process 332) so as to determine one or more individual original classification labels that make up any given combined classification label. In some embodiments, the pre-classification transformation process 312 can be configured to generate a mapping of each combined classification label reflected in the adjusted training data 318 to the corresponding one or more individual original classification labels.

[0075] In various embodiments, the multi-class classification model 320 can be a machine learning model for implementing multi-class classification for a given input data set, wherein each object of the input data set is classified into exactly one of a plurality of classification sets including at least three classification sets. In one example, a specific instance of the multi-class classification model 320 can be trained using a training data set for a specific classification task (e.g., classifying a specific type of object into a specific classification set based on specific features associated with the classified object), and this specific instance of the multi-class classification model 320 (as trained using the training data set) can then be used to generate a multi-class classification for a given input data set. The multi-label classification system 140 can be configured to train the multi-class classification model 320 based at least in part on the adjusted training data 318, and generate multi-class output data 330 for the input data set using the trained multi-class classification model 320.

[0076] More specifically, the multi-class classification model 320 can be configured by and / or at least partially based on a training process 322 and a prediction process 324 .

[0077] In various embodiments, the multi-class classification model 320 can undergo a training operation (e.g., performed by a training process 322) using a training data set. The training data set can include the adjusted training data 318 (as defined and described above) and any data that may be associated with training objects included in the adjusted training data 318 that are relevant to the specific prediction task that the multi-class classification model 320 is being trained to perform, including any portion of the enterprise data and / or the aggregated quality management data 302.

[0078] In various embodiments, an instance of the multi-class classification model 320 may include a data object created using machine learning to learn to perform a given function (e.g., prediction) by training with a training data set (e.g., adjusted training data 318). The multi-class classification model 320 may be trained to generate predictions (e.g., included in the multi-class output data 330) by learning from the training data set. For example, the multi-class classification model 320 may include a mapping function f from input variables x to discrete output variables y. The training process 322 may be configured to formulate a particular mapping function f from input variables x (corresponding to features of the training data set and the input data set associated with a particular object) to a discrete output variable y (representing a predicted set of classifications into which the particular object should be classified).

[0079] More specifically, in one example, the multi-class classification model 320 can undergo training (e.g., through a training process 322) using a training data set (e.g., including the adjusted training data 318) to identify features from the training data set and determine optimal coefficients representing adjustments or weights to apply with respect to the features to produce a target value reflected in the training data set, for example, based on positive and / or negative correlations between the target value and various features of the training data set. The target value of the multi-class classification model 320 can represent the classification of an object into a predicted classification set and can correspond to a combined classification set indicated for each training object in the adjusted training data 318 (e.g., as represented by a combined classification label). The features can correspond to various domain-specific attributes reflected in the training data set, including characteristics of training objects (e.g., text objects) included in the adjusted training data 318 and / or any attributes associated with training objects from the sample event data 306, the aggregated quality management data 302, and / or any other enterprise data stored in the enterprise data repository 304.

[0080] In some embodiments, the training data set (e.g., at least included in the adjusted training data 318) can be input into a training process 320 corresponding to the multi-class classification model 320 to train the model to generate predictions (e.g., defined in the multi-class output data 330). In one example, for the multi-class classification model 320, the result of the model training (e.g., by the training process 322) is the trained model weights 326 corresponding to the multi-class classification model 320 used by the prediction process 324. In some embodiments, after the initial training, additional training data (e.g., including subsequently received and / or generated sample event data 306) can be periodically or continuously input into the training process 322 corresponding to the multi-class classification model 320 to refine and update the model.

[0081] The multi-class classification model 320 may be trained (e.g., via a training process 322) to generate predictions (e.g., included in multi-class output data 330) based at least in part on an input data set, which may, for example, be embodied in and / or include production event data 328.

[0082] In some embodiments, the input data set (e.g., embodied in and / or including the production event data 328) used to generate the prediction (e.g., included in the multi-class output data 330) may include a current instance of any of the enterprise data and / or the aggregated quality management data 302 stored in the enterprise data repository 304 (e.g., current or latest data reflecting the moment when the prediction is being generated), and / or any other data related to the multi-class classification task being performed. In one example, the input data set (e.g., embodied in and / or including the production event data 328) may include production objects (e.g., text objects) of the same type and / or similar to the training objects reflected in the training data set (e.g., including the adjusted training data 318), including any associated attributes or features, but without any classification of the production objects to a classification set, and the prediction process 324 may be configured to generate the multi-class output data 330 to include the classification of the production objects to the classification set based on the model training performed by the training process 322 as defined and described above. In some embodiments, production event data 328 may include quality management events (e.g., from aggregated quality management data 302), including, for example, text objects containing text descriptions or narratives that, in some examples, characterize how a product is provided, consumed, and / or used, characterize a particular context in which a product is provided, consumed, and / or used, characterize certain effects resulting from the provision, consumption, and / or use of a product, among other examples, and / or represent, characterize, and / or describe customer complaints resulting from the provision, consumption, and / or use of a product subject to quality management by the quality management system 120. Production objects of an input data set may correspond to and / or include text objects from production event data 328.

[0083] In some embodiments, an input data set (e.g., embodied in and / or including production event data 328) may be input into a prediction process 324 corresponding to a multi-class classification model 320, and in response to receiving the input data set, the prediction process 324 corresponding to the multi-class classification model 320 may be configured to generate multi-class output data 330 for the input data set based at least in part on the input data set and / or trained model weights 326 generated during training of the model.

[0084] The multi-class output data 330 may include a multi-class classification of production objects from the input data set, each production object being classified into exactly one of the plurality of classification sets (i.e., a set in the combined classification sets reflected in the adjusted training data 318 used to train the model). In one example, the multi-class output data 330 may include a classification of production objects from the input data set, each production object being classified into a corresponding combined classification set in at least one combined classification set from the adjusted training data set 318.

[0085] In various embodiments, the multi-label classification system 140 can be configured to perform a multi-label classification task using the adjusted multi-class classification model 320 in combination with a post-classification processing function for transforming the multi-class classification into a multi-label classification suitable for the current enterprise management context (e.g., performing quality management operations). For example, in some embodiments, the post-classification transformation process 332 of the multi-label classification system 140 can be configured to receive the multi-class output data 330 corresponding to the multi-class classification model 320 output by the prediction process 324. The post-classification transformation process 332 can be configured to transform the multi-class output data 330 into the multi-label output data 334, for example, based at least in part on the transformation protocol 316.

[0086] In some embodiments, the multi-label output data 334 may include a multi-label classification of production objects from the input data set, each production object being classified into at least one of the multiple classification sets. In one example, the multi-label output data 334 may include a classification of production objects from the input data set, each production object being classified into at least one of the multiple original classification sets from the original training data set 310 and / or the original training data 314. In some cases, at least one of the production objects classified in the multi-label output data 334 may be classified into multiple different original classification sets.

[0087] In one example, transforming the multi-class output data 330 into the multi-label output data 334 (e.g., by the post-classification transformation process 332) may include: for each specific production object among the production objects classified in the multi-class output data 330, resolving the combined classification label corresponding to the combined classification set into which the specific production object in the multi-class output data 330 is classified into at least one original classification label corresponding to the original classification set from the original training data set 310 and / or the original training data 314 to which the specific production object in the multi-label output data 334 is to be classified. In other words, the post-classification transformation process 332 may be configured to resolve the combined classification label for each specific production object in order to determine which original classification set(s) the specific production object should be classified into and / or which original classification label(s) are assigned to the specific production object. In some embodiments, the post-classification transformation process 332 may be configured to resolve the combined classification label based at least in part on the transformation protocol 316 (including, for example, the label combiner operator defined by the transformation protocol 316). For example, the post-classification transformation process 332 can be configured to extract from each combined classification label a series of one or more individual original classification labels contained within the combined classification label based on any instances of the label combiner operator found in the combined classification label and the delimitation of the individual original classification labels within the combined classification label indicated by the placement of the instances of the label combiner operator relative to the individual original classification labels. In addition, in some embodiments, the transformation protocol 316 can define a mapping of each combined classification label reflected in the adjusted training data 318 to a corresponding one or more individual original classification labels from the original data set 310 and / or the original training data 314 (e.g., generated by the pre-classification transformation process 312 in the process of generating the adjusted training data 318). The post-classification transformation process 332 can be configured to determine the individual original classification labels to be assigned to each production object based at least in part on the mapping of the combined classification labels to the original classification labels defined in the transformation protocol 316.

[0088] In various embodiments, the multi-label classification system 140 may be configured to cause at least one enterprise management operation to be performed (e.g., by the quality management system 120) based at least in part on the multi-label output data 334. In one example, the enterprise management operations caused to be performed by the multi-label classification system 140 based on the multi-label output data 334 may include one or more quality management operations 336. For example, in some example scenarios and / or in some embodiments, the quality management system 120 may be configured to receive the multi-label output data 334 from the multi-label classification system 140 to perform quality management operations 336 (e.g., with respect to various research, development, manufacturing, distribution, and / or service provision processes associated with the production of products subject to quality management constraints of the quality management system 120). In one example, based at least in part on the multi-tag output data 334, the quality management system 120 can be configured to perform quality management operations 336, including updating the GUI of the quality management system 120 (e.g., displaying the multi-tag output data 334 and / or data derived from the multi-tag output data in a particular manner), generating and / or transmitting alerts, generating, organizing and / or formatting reports, and / or performing automated operations (e.g., generating and submitting orders, scheduling services and / or inspections, generating relevant quality management events, performing automated service and / or diagnostic operations), to name just a few examples.

[0089] Figure 4 is an illustration of example data objects used and / or produced by the multi-label classification system 140 at various stages of adapting a multi-class classification model for performing multi-label classification.

[0090] At the first stage 402, an exemplary original training data set (e.g., corresponding to and / or including the original training data set 310 and / or the original training data set 314) is shown in a form prior to being transformed into an adjusted training data set by the multi-label classification system 140. In particular, the example shown relates to classifying text objects (e.g., from quality management events) into one or more classification sets corresponding to MedDRA terms as defined and described above. The original training data set at the first stage 402 specifies values ​​corresponding to a "Description" attribute and a "MedDRA" attribute with respect to a series of training objects (e.g., text objects). The "Description" attribute may identify a particular training object (e.g., a particular text object containing a text description of a situation or condition related to a product supply produced by an enterprise), and the associated "MedDRA" attribute may indicate an original classification label associated with the training object identified via the "Description" attribute. Notably, in the exemplary original training data set shown at the first stage 402, the training objects may be associated with multiple original classification sets, e.g., various training objects have a "many-to-many" correspondence with various classification labels. In this way, for example, the original training data set shown with respect to the first stage 402 can correspond to a multi-label classification task. In the example shown, a training object identified as "Description1" is classified into both the classification set represented by the "Term1" classification label (e.g., corresponding to a specific MedDRA term) and the classification set represented by the "Term2" classification label, a training object identified as "Description2" is classified into both the classification set represented by the "Term2" classification label and the classification set represented by the "Term3" classification label, and a training object identified as "Description3" is classified only into the classification set represented by the "Term3" classification label.

[0091] At the second stage 404, an exemplary adjusted training data set (e.g., corresponding to and / or including the adjusted training data 318) is shown in the form after being transformed from the original training data set by the multi-label classification system 140 shown with respect to the first stage 402. Like the original training data set of the first stage 402, the adjusted training data set shown at the second stage 404 specifies values ​​corresponding to the "Description" attribute with respect to a series of training objects, the specified values ​​identifying a particular training object (from among those training objects indicated in the original training data set shown with respect to the first stage 402). However, now, each of the training objects is assigned a value corresponding to the "Combined MedDRA" attribute, which may represent a combined classification label (e.g., as shown with respect to Figure 3404). It is noteworthy that in the exemplary original training data set shown in the second stage 404, the training objects can be associated with exactly one combined classification set, for example, various training objects have a "one-to-one" correspondence with various classification labels. In this way, for example, the adjusted training data set shown with respect to the first stage 402 can correspond to a multi-class classification task. Each of the "combined MedDRA" attributes corresponding to the combined classification labels can represent a combination of one or more "MedDRA" attributes in the "MedDRA" attributes provided in the original training data set shown with respect to the first stage 402. For the "combined MedDRA" attributes representing a combination of multiple "MedDRA" attributes, the original "MedDRA" classification labels are all included in the corresponding "combined MedDRA" attribute delimited via the label combiner operator "_or_" (e.g., as defined in the transformation protocol) inserted between the individual original "MedDRA" labels.

[0092] In the example shown, in the second stage 404, the training object identified as "Description1" is classified into a single combined classification set represented by the "Term1_or_Term2" combined classification label due to the corresponding training object being labeled as both "Term1" and "Term2" in the original training data set. Similarly, the training object identified as "Description2" is classified into a single combined classification set represented by the "Term2_or_Term3" combined classification label due to the corresponding training object being labeled as both "Term2" and "Term3" in the original training data set. The training object identified as "Description3" is classified into a single combined classification set represented by the "Term3" combined classification label, which is unchanged from the "Term3" original classification label due to the corresponding training object being labeled as only "Term3" in the original training data set.

[0093] In the third stage 406, an exemplary multi-class classification model (e.g., corresponding to and / or including the multi-class classification model 320) is shown. The exemplary multi-class classification model is configured to generate, for each input variable X corresponding to a (training or production) object (e.g., descriptions similar to those identified in the original training dataset shown with respect to the first stage 402 and those identified in the adjusted training dataset shown with respect to the second stage 404), a corresponding output variable Z corresponding to a predicted “Combined MedDRA” combined classification label from the adjusted training dataset shown with respect to the second stage 404.

[0094] In a fourth stage 408, an exemplary mapping of combined classification labels reflected in the adjusted training data to corresponding individual original classification labels is provided, which can be used to transform the multi-class output data generated by the model shown with respect to the third stage 406 into multi-label output data. In various embodiments, the mapping can be generated based on the set of combined classification labels reflected in the adjusted training data set, for example, in conjunction with transforming the original training data set into the adjusted training data set, or in conjunction with transforming the multi-class output data into multi-label output data. In one example, the mapping can be generated based at least in part on a transformation protocol, and in particular at least in part on a label combiner operator defined in the transformation protocol, for example, by parsing the "combined MedDRA" combined classification labels with respect to the label combiner operator to determine which "MedDRA" original classification labels correspond to each "combined MedDRA" combined classification label. In the mapping, each “Combined MedDRA” combined classification label is associated with a binary value corresponding to each specific “MedDRA” raw classification label from the original training data set (“Term1”, “Term2”, “Term3”), wherein a first binary value “1” indicating that the corresponding “MedDRA” raw classification label is mapped to the associated “Combined MedDRA” combined classification label, and a second binary value “0” indicating that the corresponding “MedDRA” raw classification label is not mapped to the associated “Combined MedDRA” combined classification label.

[0095] In the example shown, the "Term1_or_Term2" combined classification label is mapped to the "Term1" original classification label and the "Term2" original classification label, but not to the "Term3" original classification label. Therefore, any object (e.g., a production object from an input data set) assigned with the "Term1_or_Term2" combined classification label in the multi-class output data will be assigned both the "Term1" original classification label and the "Term2" original classification label in the corresponding multi-label output data (which is generated by transforming the multi-class output data). Similarly, the "Term2_or_Term3" combined classification label is mapped to the "Term2" original classification label and the "Term3" original classification label, but not to the "Term1" original classification label. Therefore, any object assigned with the "Term2_or_Term3" combined classification label in the multi-class output data will be assigned both the "Term2" original classification label and the "Term3" original classification label in the corresponding multi-label output data. Finally, the "Term3" combined classification label is only mapped to the "Term3" original classification label. Therefore, any object assigned the “Term3” combined classification label in the multi-class output data will only be assigned the “Term3” original classification label in the corresponding multi-label output data.

[0096] Having described the exemplary systems and / or devices of the present disclosure, exemplary flowcharts including various operations performed by the devices and / or systems described herein will now be discussed. It should be understood that each flowchart in the flowchart depicts an exemplary computer-implemented process that can be performed by one or more of the devices, systems and / or equipment described herein, such as by using one or more of its components. The boxes indicating the operation of each process can be arranged in any of a variety of ways, as depicted and described herein. In some such embodiments, one or more boxes of any process in the process described herein occur between one or more boxes of another process, before one or more boxes of another process, and / or otherwise operate as a subprocess of a second process. In addition or alternatively, any process in the process may include some or all of the steps described and / or depicted, including one or more optional operation boxes in some embodiments. With respect to the flowcharts discussed below, in some or all embodiments of the present disclosure, one or more boxes in the depicted boxes may be optional. Similarly, it should be understood that one or more operations in the operations of each flowchart may be combinable, replaceable and / or otherwise changed, as described herein.

[0097] Figure 5 A flowchart including operational blocks of exemplary processes according to at least some exemplary embodiments of the present disclosure is shown. In some embodiments, Figure 5 The computer-implemented process of is embodied by a computer program code stored on a non-transitory computer-readable medium of a computer program product, the computer program code being configured to be executed to perform the computer-implemented method. Alternatively or in addition, in some embodiments, Figure 5 The exemplary processes of are performed by one or more specially configured computing devices (such as specially configured apparatus 200) (e.g., via transformation circuit 210, training circuit 212, classification circuit 214, and / or integrated circuit 216). In this regard, in some such embodiments, apparatus 200 is specifically configured by computer program instructions stored on the apparatus, such as stored in memory 204 and / or another component depicted and / or described herein and / or otherwise accessible to apparatus 200, for performing operations with respect to Figure 5 In some embodiments, the specially configured device 200 includes and / or otherwise communicates with one or more external devices, systems, equipment, etc. to perform one or more of the operations depicted and described. Figure 5 The operation blocks of the exemplary processes are depicted in a particular order, but this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed.

[0098] Figure 5 A flow diagram is shown including operational blocks of an example process 500 for performing multi-label classification using a tuned multi-class classification model, in accordance with at least some example embodiments of the present disclosure.

[0099] Process 500 begins at operation 502, where a device (such as, but not limited to, Figure 2 The apparatus 200 or circuit thereof described herein transforms an original training data set corresponding to a multi-label classification task into an adjusted training data set corresponding to a multi-class classification task based at least in part on a predefined transformation protocol. In various embodiments, the original training data set transformed at operation 502 may correspond to and / or include the following: Figure 3 The original training data set 310 and / or the original training data 314 defined and described, the adjusted training data set generated at operation 502 may correspond to and / or include the original training data set 310 and / or the original training data set 314 as described with respect to FIG. Figure 3 The adjusted training data 318 and / or any training data set including the adjusted training data 318 defined and described, and transforming the original training data set into the adjusted training data set at operation 502 may include, for example, Figure 3 Some or all of the functionality described may be attributed to the similar functionality of the pre-classification transformation process 312. In addition or alternatively, the predefined transformation protocol referenced with respect to operation 502 may correspond to and / or include the predefined transformation protocol referenced with respect to operation 502. Figure 3 Transformation protocol 316 defined and described.

[0100] At operation 504 of process 500, a device (such as, but not limited to, the above-mentioned Figure 2 The apparatus 200 or circuit thereof described herein trains a multi-class classification model based at least in part on the adjusted training data set generated at operation 502. In various embodiments, the multi-class classification model trained at operation 504 may correspond to and / or include the following: Figure 3 The multi-class classification model 320 defined and described, and training the multi-class classification model at operation 504 may include, for example, Figure 3 Some or all of the similar functionality described is attributed to the training process 322.

[0101] At operation 506 of process 500, a device (such as, but not limited to, the above-mentioned Figure 2 The apparatus 200 or circuit thereof described herein generates multi-class output data for the input data set using the multi-class classification model trained at operation 504. In various embodiments, the input data set referenced with respect to operation 506 may correspond to and / or include the data set referenced with respect to operation 506. Figure 3 The production event data 328 and / or any input data set including the production event data 328 defined and described, the multi-category output data generated at operation 506 may correspond to and / or include the following: Figure 3 The multi-category output data 330 defined and described, and generating the multi-category output data 330 at operation 506 may include, for example, Figure 3 Some or all of the similar functionality described is attributed to prediction process 324.

[0102] At operation 508 of process 500, a device (such as, but not limited to, the above-mentioned Figure 2 The apparatus 200 or its circuit described herein) transforms the multi-class output data generated at operation 506 into multi-label output data corresponding to a multi-label classification task (refer to operation 502) based at least in part on a predefined transformation protocol (refer to operation 502). In various embodiments, the multi-label output data generated at operation 508 may correspond to and / or include the following: Figure 3 The multi-label output data 334 defined and described, and generating the multi-label output data 334 at operation 508 may include, for example, Figure 3 Some or all of the similar functionality described is attributed to the post-classification transformation process 332.

[0103] At operation 510 of process 500, a device (such as, but not limited to, the above-mentioned Figure 2The apparatus 200 or circuitry thereof described herein causes at least one enterprise management operation to be performed based at least in part on the multi-tag output data generated at operation 508. In various embodiments, the enterprise management operation caused to be performed at operation 510 may correspond to and / or include the following: Figure 3 Define and describe quality management operations336.

[0104] Although exemplary processing systems have been described in the text and figures, specific implementations of the subject matter and functional operations described herein may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0105] The embodiments of the subject matter and operations described herein may be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described herein may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions) encoded on a computer storage medium for execution by an information / data processing device or for controlling the operation of an information / data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal, or electromagnetic signal), which is generated to encode information / data for transmission to a suitable receiver device for execution by an information / data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. The computer storage medium may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0106] The operations described herein may be implemented as operations performed by an information / data processing apparatus on information / data stored on one or more computer-readable storage devices or received from other sources.

[0107] The term "data processing device" covers all kinds of devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality of the foregoing devices or a combination thereof. The device may include a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)). In addition to hardware, the device may also include code that creates an execution environment for the computer program under consideration (e.g., code constituting processor firmware, a protocol stack, a repository management system, an operating system, a cross-platform operating environment, a virtual machine, or a combination of one or more of them). The device and the execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing infrastructures, and grid computing infrastructures.

[0108] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or information / data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute the computer program on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0109] The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input information / data and generating output. By way of example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors and any one or more processors of any kind of digital computer. Generally speaking, the processor will receive instructions and information / data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memories for storing instructions and data. Generally speaking, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to the one or more mass storage devices to receive information / data from the one or more mass storage devices or transfer information / data to the one or more mass storage devices, or both. However, a computer does not need to have such devices. Devices suitable for storing computer program instructions and information / data include all forms of non-volatile memory, media, and storage devices, including by way of example semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0110] To provide interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information / data to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball through which the user can provide input to the computer). Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from a web browser on a user's client device.

[0111] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component (e.g., as an information / data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or web browser through which a user can interact with a specific implementation of the subject matter described herein), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital information / data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnections (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0112] The computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is obtained by means of a computer program running on each computer and having a client-server relationship with each other. In some embodiments, the server transmits information / data (e.g., an HTML page) to a client device (e.g., for displaying information / data to a user interacting with the client device and receiving user input from a user interacting with the client device). Information / data (e.g., the result of user interaction) generated at the client device can be received from the client device at the server.

[0113] Although this specification includes many specific implementation details, these details should not be interpreted as limiting the scope of any disclosed or claimable content, but should be interpreted as a description of specific features for a specific disclosed specific embodiment. Certain features described herein in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a separate embodiment may also be implemented in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as acting in some combinations and even initially claimed, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for a sub-combination or a variation of the sub-combination.

[0114] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated together in a single software product or grouped into multiple software products.

[0115] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some specific implementations, multitasking and parallel processing may be advantageous.

[0116] It should be understood that the disclosure is not limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation unless otherwise specified.

Claims

1. An apparatus comprising at least one processor and at least one non-transitory memory, the at least one non-transitory memory including program code stored thereon, wherein the at least one non-transitory memory and the program code are configured to, with the at least one processor, cause the apparatus to at least: transforming an original training dataset corresponding to a multi-label classification task into an adjusted training dataset corresponding to a multi-class classification task based at least in part on a predefined transformation protocol, wherein a multi-class classification model configured to perform the multi-class classification task is trained based at least in part on the adjusted training dataset; transforming multi-class output data generated for an input dataset using a trained multi-class classification model into multi-label output data corresponding to the multi-label classification task based at least in part on the predefined transformation protocol; and At least one enterprise management operation is caused to be performed based at least in part on the multi-tag output data.

2. The apparatus according to claim 1, wherein the original training data set comprises a multi-label classification of training objects, each training object is classified into at least one original classification set of a plurality of original classification sets, and wherein at least one of the training objects is classified into a plurality of different original classification sets.

3. The apparatus according to claim 2, wherein the adjusted training dataset comprises a multi-category classification of the training objects, each training object being classified into a corresponding combined classification set in at least one combined classification set, wherein the combined classification set corresponding to each specific training object among the training objects represents a combination of each original classification set to which the specific training object is classified in the multi-label classification of the original training dataset.

4. The apparatus of claim 3, wherein the multi-class output data comprises a classification of production objects from the input dataset, each production object being classified into a corresponding combined classification set of the at least one combined classification set from the adjusted training dataset.

5. The apparatus of claim 1 , wherein the at least one non-transitory memory and the program code are configured to, with the at least one processor, further cause the apparatus to at least: The multi-class classification model is trained based at least in part on the adjusted training dataset.

6. The apparatus of claim 1 , wherein transforming the original training dataset into the adjusted training dataset comprises: For each specific training object among the training objects classified in the original training data set, all original classification labels corresponding to the original classification set to which the specific training object in the original training data set is classified are combined into a combined classification label, wherein the combined classification label includes all the original classification labels for the specific training object delimited by a label combiner operator defined according to the predefined transformation protocol.

7. The apparatus of claim 1 , wherein transforming the multi-category output data into the multi-label output data comprises: For each specific production object among the production objects classified in the multi-category output data, at least in part based on a label combiner operator defined according to the predefined transformation protocol, a combined classification label corresponding to the combined classification set into which the specific production object in the multi-category output data is classified is parsed into at least one original classification label corresponding to the original classification set from the original training data set into which the specific production object in the multi-label output data will be classified, wherein the combined classification label includes the at least one original classification label delimited by the label combiner operator.

8. The apparatus of claim 1, wherein the multi-label classification task comprises classifying a text object containing text characterizing a quality management event associated with a product produced by an enterprise into a classification set corresponding to medical terms from a medical dictionary.

9. A computer-implemented method, comprising: transforming an original training dataset corresponding to a multi-label classification task into an adjusted training dataset corresponding to a multi-class classification task based at least in part on a predefined transformation protocol, wherein a multi-class classification model configured to perform the multi-class classification task is trained based at least in part on the adjusted training dataset; transforming multi-class output data generated for an input dataset using a trained multi-class classification model into multi-label output data corresponding to the multi-label classification task based at least in part on the predefined transformation protocol; and At least one enterprise management operation is caused to be performed based at least in part on the multi-tag output data.

10. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions including an executable portion, the executable portion configured to: transforming an original training dataset corresponding to a multi-label classification task into an adjusted training dataset corresponding to a multi-class classification task based at least in part on a predefined transformation protocol, wherein a multi-class classification model configured to perform the multi-class classification task is trained based at least in part on the adjusted training dataset; transforming multi-class output data generated for an input dataset using a trained multi-class classification model into multi-label output data corresponding to the multi-label classification task based at least in part on the predefined transformation protocol; and At least one enterprise management operation is caused to be performed based at least in part on the multi-tag output data.

Images